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I.     INTRODUCTION 

A.      GENERAL 

The  search  for  better  ways  to  do  things  in  government  is  not  new. 
However,   in  Naval  ship  acquisition,   a  major  attempt  at  better  decision- 
making is  developing. 

Both  weapons  and  systems  for  delivering  them  have  gone  through 
several  revolutions  in  the  years  since  the  end  of  the  Second  World  War. 
Nuclear  bombs  are  now  thousands  of  times  more  powerful  than  those 
dropped  on  Hiroshima  and  Nagasaki.     Breakthroughs  in  missile  tech- 
nology are  threatening  existing  systems.     Men  in  space  have  provided  for 
military  possibilities  not  thought  of  two  decades  ago.     Keeping  ahead  in 
the  technological  race  is  not  in  itself  a  guarantee  of  security  in  these 
circumstances.     It  remains  essential  to  incorporate  developing  tech- 
nology into  operational  hardware  and  to  deploy  and  use  this  hardware 
with  skill  and  intelligence.     No  amount  of  production  skill  or  intelligent 
use  can  compensate  for  significant  shortcomings  in  the  development,    test, 
and  evaluation  of  the  emerging  hardware.      These  important  first  steps 
will  be  explored  in  this  thesis. 

In  parallel  with  the  technological  revolution  in  weapon  systems  in 
the  past  few  decades,    we  have  seen  the  costs  of  building  and  maintaining 
U.   S.    Naval  ships  multiply  to  astounding  heights.      The  magnitude  of  these 
costs  is  emphasized  by  the  recognition  that  overhaul  and  modernization  of 
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computerized  missile  ships  now  is  often  more  expensive  than  procure- 
ment costs  of  ships  in  the  past.     In  the  fiscal-year  1975  Navy  budget  of 
$29.6  billion  dollars,    over  $3.6  billion  is  allocated  to  the  procurement 
of  six  classes  of  Navy  vessels.     The  process  of  procuring  ships  is  the 
longest,    most  complex  and  greatest  in  dollar  volume,    of  any  in  the 
Defense  Department.      The  cycle  from  concept  to  delivery  of  a  new  ship 
may  require  five  to  ten  years,    with  technical  improvements  evolving 
the  whole  time. 

Weapon  systems  procurement  functions  are  located  at  the  highest 
organizational  levels  in  the  Services  and  involve  concentrations  of  the 
top  technical,    procurement,    test,    and  evaluation  talent.      This  has  been 
driven  by  an  iterative  process  whereby  the  accelerating  state  of  the  art 
has  driven  the  design  methodology,    management  requirements,   and  sub- 
sequently the  costs.     Program  management  has  been  continually  revised 
in  an  effort  to  keep  stride  with  this  process  and  ensure  the  best  ships 
possible  while  keeping  effective  cost  control.      Thus  the  process  des- 
cribed in  this  thesis  consumes  major  portions  of  the  resources  of  the 
Navy. 

Previous  to  World  War  II,    the  Navy  did  much  of  its  own  design  and 
development  and  even  some  production.      But  after  the  war  the  Navy  lost 
much  of  this  in-house  capability.      Consequently,    the  Navy  has  of  neces- 
sity become  increasingly  reliant  upon  industry  for  almost  every  aspect 
of  weapon  design,    development,    test,    evaluation,    and  production.      The 
Navy  has  shifted  its  thinking  over  the  years  from  an  absolute  autocratic 


1    * 


control  of  this  process  in  an  internal  environment,    to  a  more  tenuous 
and  democratic  committee  control  in  a  largely  external  contractor 
environment  --  one  that  is  much  more  in  the  public  eye.     Unfortunately, 
the  Navy  hasn't  made  this  transition  smoothly.     Within  this  context,    one 
area  which  has  not  kept  pace  and  is  sorely  in  need  of  a  quantum  improve- 
ment is  the  development  and  management  of  test  and  evaluation  programs. 
These  should  be  structured  to  assist  top  management  in  making  procure- 
ment decisions  and  to  reduce  the  uncertainties  surrounding  these  decisions. 

Test  and  evaluation  of  proposed  new  weapon  systems  is  one  of  the 
Navy's  key  controls  in  the  complex  process  of  acquiring  today's  multi- 
billion  dollar  systems.      Testing  at  decisive  stages  of  development  shows 
where  problems  exist  and  helps  Navy  managers  to  make  better  decisions 
affecting  future  production  and  purchases  of  weapons  than  would  other- 
wise be  possible.     The  test  and  evaluation  of  emerging  systems  within 
itself  involves  uncertainty  however.      "How  much  is  enough?  "   must  be 
answered  for  testing  itself.     It  is  apparent  that  many  of  the  problems  of 
managing  a  test  and  evaluation  program  have  important  economic  aspects. 

The  fundamental  problem  of  how  much  to  spend  on  a  test  and  evalua- 
tion program  is  an  allocation  problem  within  the  ship  acquisition  program. 
Economic  theory  tells  us  that  we  "should"  continue  to  spend  on  test  and 
evaluation  until  the  marginal  gain  from  the  expenditure's  is  just  equal  to 
the  gain  from  expenditures  elsewhere.      But  in  the  case  of  test  and  evalua- 
tion,   this  formula  is  peculiarly  difficult  to  interpret  or  apply.      The  gain 
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is  much  more  uncertain  --  much  harder  to  predict  with  accuracy  --  than 
the  gain  from,    say,   an  additional  fighter  defense  squadron  with  aircraft 
of  known  performance.     This  is  particularly  true  of  developmental  test 
and  evaluation,    where  the  product,    if  any,    will  be  mostly  knowledge  -- 
and  knowledge  usually  far  removed  from  its  practical  end  use.     Rather, 
its  purpose  is  one  of  facilitating  the  evolution  of  the  ship  system  itself. 
Rapid  advances  in  electronics  and  computer  technology  have  had  a 
profound  effect  on  the  complexity  of  shipboard  systems  and  the  test  and 
evaluation  needed  to  support  the  procurement  of  these  systems.      There 
is  a  definite  need  in  the  Fleet  for  ever-increasing  speed  and  accuracy  in 
shipboard  data  processing  to  meet  more  sophisticated,    high-speed  threats, 
As  a  result,    the  key  weapon  and  sensor  systems  are  utilizing  digital  inter. 
faces  and  processing  to  as  great  an  extent  as  possible.      The  cost  and 
complexity  of  the  equipment  and  a  movement  toward  human- engineered 
ships  does  not  allow  us  to  keep  the  luxury  of  back-up  and  casualty  mode 
operation  for  many  of  these  functions  that  we  had  formerly  enjoyed.     The 
demands  for  greater  maintainability  and  reliability  are  also  forcing  in- 
creased digitalization.      The  result  is  a  level  of  hardware  and  software 
integration  never  previously  encountered  in  a  ship  system. 

B.      THE  PATROL  FRIGATE  PROGRAM 

The  Patrol  Frigate  (PF  109)   Class,    for  example,    will  have  digital 
fire  control  systems  in  the  missile,    gun  and  antisubmarine  warfare 
areas,    all  linked  to  command  and  control  within  a  centrally-located 
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computer  complex.     To  complicate  matters  further,    at  the  time  the  PF 
program  was  being  formulated,    DOD  policy  regarding  major  weapon- 
systems  development,    acquisition  and  test  was  being  drastically  changed. 
In  a  memorandum  issued  in  the  Spring  of  1970,    Deputy  Secretary  of 
Defense  David  Packard  indicated  that  experience  with  the  Total  Package 
Procurement  approach  to  development  and  acquisition  of  major  systems 
was  unsatisfactory;  that  it  should  be  used  with  caution;  and  that  a  new 
policy  would  soon  be  established.      The  new  policy  took  the  form  of  DOD 
Instruction  5000.  1  which  outlined  the  "Fly-Before-Buy"  approach. 
Essentially  this  new  approach  requires  the  services  to  develop  and  dem- 
onstrate the  operability  of  a  weapons  system  by  actual  test  prior  to  the 
commitment  of  funds  for  mass-production  of  any  major  system.      Progress 
in  meeting  this  requirement  is  reviewed  by  the  Defense  Systems  Acquisi- 
tion Review  Council  (DSARC)  at  discrete  stages  during  development. 
DSARC  determines  whether  the  system  is  ready  to  advance  to  the  next 
stage.      It  then  makes   recommendations  to  the  Secretary  of  Defense  con- 
cerning funding  of  the  next  stage  and  follow- on  procurement. 

The  very  design  of  the  PF  evolved  in  parallel  with  emerging  "Fly- 
Before-Buy"  policies  within  DOD.     At  the  CNO/SECNAV  level  it  was 
soon  realized  that  it  was  impractical  from  a  cost  and  procurement 
schedule  basis  to  adhere  to  the  letter  of  the  law  of  emerging  policies. 
Tradeoffs  were  made  such  that  adequate  initial  test  and  evaluation  for 
subsystems  is  to  be  demonstrated  prior  to  large-scale  ship  production. 
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Additionally,    critical  ship  system  integration  is  to  be  demonstrated  at 
shore-based  test  sites.     In  effect  then,    we  have  swung  from  "Total 
Package  Procurement"  to  "Fly-Before-Buy"  with  fewer  paper  solutions 
and  more  actual  hardware  testing,   hardware  proofing  and  prototyping  at 
the  system,    subsystem,    and  component  level. 

As  early  as  the  concept  exploration  phase  of  the  PF  procurement, 
it  was  realized  that  weapon  selection  and  subsequent  test  and  evaluation 
would  be  a  controlling  influence  on  the  eventual  size  as  well  as  on  the 
ship  cost  of  the  PF.      The  Ship  Acquisition  Project  Manager  (SHAPM), 
PMS  399,   has  identified  the  integration  of  the  combat  system  as  a  critical 
task  in  the  PF  acquisition  effort.     In  conjunction  and  compliance  with 
emerging  DOD  test  and  evaluation  requirements,    the  PF  program  has 
been  structured  specifically  to  provide  for  an  extensive  test  program 
which  should  substantially  reduce  technical,    cost,    and  schedule  risks. 
The  information  obtained  from  an  evaluation  of  combat- system  test  data 
will  be  a  major  factor  in  the  final  decisions  on  procurement  of  follow- ships 
at  CEB,    DSARC,    and  Congressional  levels. 

Because  of  its  critical  nature,    the  combat  system  integration  effort 
will  be  decoupled  from  the  time-critical  events  of  the  shipyard  and  con- 
ducted at  a  land-based  test  site.      The  land-based  test  site  will  contain 
an  installation  duplicating,    insofar  as  possible,    the  shipboard  installation 
of  all  sensors,    computing  equipment,    weapons  and  related  subsystems. 
Interface  equipment  will  be  included.     Major  factors  in  achieving  full 
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utilization  of  the  integrated  system  which  are  to  be  demonstrated  at  the 
land-based  test  site  are  envisioned  to  be  computer  program  debugging 
and  physical,    electrical  and  mechanical  compatibility.     Both  of  these  are 
traditionally  troublesome  areas  in  complex  ships. 

C.     SCOPE  OF  THESIS 

The  major  objectives  of  this  thesis  were:    to  investigate  and  analyze 
emerging  weapon  system  procurement,    test,   and  evaluation  policies 
within  DOD  and  the  Navy  Department;  to  document  efforts  within  the  PF 
development  process  in  view  of  these  policies;  and  to  gain  insight  into  the 
use  of  a  combat  system  land-based  test  site  in  implementing  these  pol- 
icies in  the  procurement  cycle. 
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II.     FROM  ROCKS  TO  COMBAT  SYSTEMS 

A.      BACKGROUND 

Man,    in  his  desire  for  a  quicker  means  of  destroying  an  enemy's 
capacity  for  waging  war,    has  continually  increased  the  complexity  and 
destructive  force  of  the  weapons  in  his  arsenal.     From  the  era  of  the 
rock-throwing  caveman,    weaponry  has  now  evolved  to  the  era  of  nuclear- 
powered  ships  and  sophisticated  guided  missiles.      Early  in  the  history 
of  the  Navy,    the  importance  of  forward-looking  weapon  system  acquisition 
was  recognized  by  the  organizer  of  the  Navy  Gun  Factory,    John  Adolphus 
Dahlgren,    when  he  stated,    "If  the  Navy  be,    indeed,    the  right  arm  of 
defense,    her  guns  and  ordnance  .  .  .   are  the  main  sinews  and  arteries, 
the  neglect  of  which  soon  render  it  feebled  and  palsied.  " 

Rightly  called  the  "father  of  naval  ordnance,  "  Dahlgren  revolutionized 
the  science  of  ordnance  with  his  original  work  and  set  standards  for  other 
navies.     In  1850  he  urged  construction  of  frigates  armed  entirely  with 
heavy  guns,    anticipating  the  dreadnaughts  of  the  next  century.     A  blue- 
water  sailor  as  well  as  a  scientist,   Admiral  Dahlgren  brilliantly  com- 
manded a  powerful  naval  force  through  two  years  of  arduous  wartime 
service  during  the  Civil  War,    leaving  behind  him  an  example  of  produc- 
tive study  and  gallantry  in  action. 


Memoirs  of  John  A.    Dahlgren,    1891,    by  Madelin  V.    Dahlgren. 


'    v 


Why  the  history?      The  point  is  that  in  the  early  days  of  weapon 
acquisition,    test,    evaluation  and  operational  use,    it  was  not  all  that 
unusual  for  Navy  personnel  to  recognize  a  need  for  an  improvement  or 
new  weapon,    to  design  and  manufacture  the  weapon  within  the  Navy 
establishment,    and  for  the  conceivers  and/or  producers  to  take  the 
weapon  to  sea  for  test  and  evaluation.     If  a  deficiency  in  ordnance  design 
was  noted,    it  was  often  corrected  on  the  spot  by  shipboard  gunners. 

To  a  certain  extent,    a  similar  path  was  followed  in  the  introduction 
of  electronic  devices  to  naval  warships.     Radio  was  developed  as  a  result 
of  the  intense  desires  of  men  to  exchange  intelligence  as  rapidly  as 
possible  over  long  distances  without  physical  interconnections.     Until 
the  present  century,    a  ship's  isolation  was  complete  once  she  navigated 
out  of  port  and  sailed  over  the  horizon.     Of  course,    ships  in  company  or 
in  passing  "spoke"  to  each  other  by  voice  hail,    or  signal  flags  by  day, 
and  lanterns  at  night.     But  once  at  sea,    orders  or  instructions  from 
higher  commands  not  in  company  could  not  be  easily  altered.      For  want 
of  better  communications,    major  battles  have  been  fought  after  peace 
treaties  were  signed. 

The  U.   S.    Navy,    in  keeping  with  its  traditional  scientific  leadership, 
early  recognized  the  impact  of  radio  on  naval  operations.      Only  by  such 
means  of  communication  could  far- scattered  forces  be  effectively 
directed.     Admiral  Bradley  A.    Fiske,    while  a  lieutenant,    experimented 
with  "electronic  communication"  on  naval  vessels  in  1888,    many  years 
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before  Marconi's  successful  application.     During  Fiske's  investigations, 
he  discovered  the  principles  of  degaussing  ships  as  a  protection  against 
mines  (used  widely  in  World  War  II),    and  designed  a  system  of  radio 
control  of  torpedoes  which  forms  the  basis  for  the  modern  "radio" 
guided  missile.     Thus  in  the  early  days  of  electronics,    Navy  personnel 
often  followed  advanced  concepts  from  a  glimmering  thought  to  useful 
hardware  aboard  ships. 

Prior  to  World  War  II,    ordnance,    electronics  and  communications 
were  more  or  less  separate  and  individual  functions  aboard  ship.     A  new 
gun  or  electronic  (radio)  device  could  be  developed,    tested,    evaluated, 
and  placed  in  an  operational  status  aboard  ship  in  relative  isolation. 
That  is,    weapons  were  weapons,    and  radios  were  radios,    and  there  was 
little  marriage  between  them. 

During  the   1930 's  the  governments  of  Great  Britain,    the  United 
States,    and  Germany  fostered  experiments  to  create  a  reliable  detector 
that  would  locate  airplanes  not  visible  to  the  naked  eye.      The  sound- 
detectors  that  had  been  used  for  this  purpose  had  become  almost  use- 
less when  the  increasing  speed  of  airplanes  made  it  necessary  for  the 
gunners  to  "lead"  their  direction  indicators  by  20  degrees  or  more  when 
firing.     Although  all  three  of  the  countries  named  were  active  in  the 
development  of  radar,    Great  Britain  led  because  it  had  the  most  to 
fear  from  air  attacks. 

Radar  was  soon  integrated  with  the  fire  control  function  of  gunnery, 
since  it  provided  a  more  sure  method  of  determining  ranges  than  optical 
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instruments  and  worked  regardless  of  night,    fog,    or  distance.     The 
marriage  of  radar  to  gunnery  in  the  area  of  fire  control  and  the  increas- 
ing swiftness  of  naval  engagements,    especially  sea-air  battles,    brought 
into  existence  a  new  organization,    the  CIC  or  combat  information  center, 
on  warships.     The  CIC  filtered  out  the  multitude  of  items  of  information 
that  came  in  from  the  radar,    spotting  planes,    sonar,    and  so  on,    reducing 
them  to  manageable  proportions  to  enable  the  commanding  officer  to 
make  intelligent  decisions. 

Supersonic  jets,    greater  threats  from  other  surface  units,    and 
advancements  in  submarine  warfare  soon  overran  the  man-machine 
interface  present  in  CIC  during  much  of  World  War  II  and  the  Korean 
War.      Computers  were  introduced  as  an  aid  to  commanders  in  their 
information  gathering  and  decision-making  processes.     Application  of 
the  computer  in  naval  weapons  systems  has  advanced  to  the  point  where 
man-machine  interface  has  been  minimized  to  the  point  where  in  many 
instances  human  resources  are  no  longer  required. 

Throughout  the  course  of  history,    the  development  of  weapons  has 
been  extremely  slow.     It  is  said  that  until  1000  A.D.,    weapons  had  a 
useful  life  of  about  400  years.      Today,    the  useful  life  span  of  a  weapon 
system  may  only  be  from  3  to  7  years  before  rendered  obsolete  by 
emerging  technology  or  enemy  advances.     The  design,    development, 
test  and  evaluation  phases  of  weapon-system  acquisition  have  become 
increasingly  important  as  the  useful  life  span  of  weapons  has  decreased. 
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It  is  the  speed  of  development,    test  and  evaluation  which  largely  deter- 
mines the  extent  to  which  the  system  design  incorporates  the  latest 
technical  advances. 

B.      WEAPON  SYSTEM  CONCEPTS 

The  concept  of  a  weapon  system  has  come  into  being  rather  recently 
in  naval  history,    evolving  from  ordnance,    radio,    radar  and  fire  control 
to  the  point  where  a  weapon  system  is  made  up  of  a  number  of  unique, 
specialized  components  which  must  be  integrated  to  achieve  overall 
effectiveness.      Ordnance  Pamphlet  (OP)  3000  defines  a  weapon  system 
as  "a  collection  of  integrated  components  or  subsystems  which  perform 
the  interrelated  functions  necessary  to  render  the  desired  effect  on  the 
enemy.  "     Recent  thinking  has  further  integrated  weapon  systems  into 
naval  ships  by  dividing  a  ship  into  a  containment  system  and  a  combat 
system.     The  containment  system  is  viewed  as  the  ship  structure  and 
related  housekeeping  systems,    e.  g.  ,    the  hull,    damage  control,    laundry 
and  messes.      The  combat  system  is  thought  of  as  being  the  command 
and  control,    navigation,    external  communications,    weapon  and  other 
supporting  systems  used  directly  for  target  surveillance,    target  recogni- 
tion,   electronic  countermeasures,    weapon  delivery,    weapon  handling 
and  storage,    and  weapon  safety.      The  complete  combat  system  does 
not  consist  of  the  physical  equipment  alone,    but  also  of  technical  manuals, 
training  plans  and  equipment,    spare  parts,    of  an  optimum  employment 
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doctrine,    and  the  human  resources  necessary  to  pursue  objectives. 
Weapons  system  procurement  thus  includes  concurrent  development, 
test  and  evaluation  of  all  of  these  items. 
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III.     EVOLUTION  OF  TEST  fc  EVALUATION  CONCEPTS 

A.      UP  TO  McNAMARA  AND  1961 

From  the  perspective  of  Test  and  Evaluation,    the  weapon  acquisition 
strategies  which  evolved  within  the  Navy  prior  to  the  appointment  of 
Secretary  McNamara  in  1961  suffered  from  a  number  of  problems  which 
paralleled  those  of  planning  and  budgeting.      For  example,    funds  for 
adequate  testing  were  often  not  available  and  test  results  were  seldom 
available  in  time  to  influence  the  major  program  decisions  which  shaped 
the  character  and  quantity  of  systems.     Procurement  of  weapon  systems 
was  production  orientated,    with  test  and  evaluation  taking  the  back  seat 
to  hardware.      Each  major  weapon  subsystem  was  tested  individually 
prior  to  shipboard  deployment.      The  first  time  all  interacting  subsystems 
were  tested  as  a  single  functioning  system  was  after  shipboard  installation. 
Increasing  system  complexity  and  the  requirement  for  the  combat  system 
to  function  under  semi-automatic  central  control  in  a  shipboard  environ- 
ment was  not  mirrored  in  total  system  test  and  evaluation  concepts. 

In  conjunction  with  shortcomings  in  test  and  evaluation  concepts, 
the  government  in  the  late  50's  and  early  60's  regarded  military  procure- 
ment efforts  as  very  risky  and  therefore  used  the  cost-plus-fixed-fee 
type  of  contract  for  this  work.     During  this  period  centralized  govern- 
mental planning  and  direction  of  military  procurement  programs  were 
lacking.     As  a  result,    many  duplicative  approaches  were  funded  to 
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meet  poorly  conceived  military  requirements.     Programs  were  initiated 
without  adequate  evaluation  of  the  time  and  cost  required  to  complete 
them,    and  performance  and  reliability  requirements  were  often  left  to 
the  contractors  to  define  after  contract  awards  had  been  made.     In  this 
atmosphere,    schedule  slippages  and  cost  overruns  of  200%  to  300%  were 
typical,    and  many  programs  were  cancelled  before  completion  because 
of  infeasibility,    obsolescence,    or  excessive  cost. 

Elements  of  each  service  sought  funding  for  programs  that  would 
enhance  their  own  military  role.      Funds  usually  being  in  short  supply, 
the  procuring  service  would  often  encourage  the  contractor  to  bid  un- 
realistically  low  by  revealing  the  amount  of  money  available  for  the 
project.     In  an  effort  to  get  hardware  out  of  the  factory  and  aboard  ship, 
test  and  evaluation  programs  were  often  shortened  or  completely  chopped 
from  the  procurement  cycle.     It  remained  for  untrained  sailors  on  board 
ship  to  discover  that  the  weapon  systems  fell  short  of  filling  the  require- 
ment for  which  they  were  originally  procured. 

The  nature  of  the  contractors  with  whom  the  services  were  dealing 
also  were  going  through  changes.     In  years  past,    the  Navy  dealt  with 
contractors  who  were  actually  producers.      These  were  real  technical 
people  with  responsibility  but  who  were  on  a  first  name  basis  with  Navy 
personnel.      These  people  had  been  at  sea  and  knew  the  bow  from  the 
stern  of  a  ship.      The  Navy  could  expect  technical  performance  from 
men  like  this  who  felt  a  strong  personal  pride  in  any  equipment  they 
provided. 
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During  this  time  period,    for  a  variety  of  complex  reasons,    most 
defense  contractors  were  moving  toward  conglomerate  corporations. 
The  Navy  no  longer  talked  with  owner-operators.     They  talked  to  cor- 
porate executives  who  often  knew  little  about  building  ships  and  weapon 
systems  but  were  interested  in  cash  flow  and  return  on  investment.      They 
were  often  lacking  in  detailed  technical  knowledge  of  weapons,   having 
reached  their  management  position  through  other  channels,    but  were  very 
good  about  discussing  concepts  such  as  schedules  and  costs.      They  may 
have  known  the  cost  impact  of  a  proposed  change,    but  they  didn't  know  the 
other  ramifications. 

Since  the  contracts  were  the  cost-plus -fixed-fee  types  and  the  con- 
tractor was  assured  of  recovering  his  full  cost,    the  contractor  often  was 
willing  to  bid  well  below  the  probable  final  cost  in  competing  for  the 
contract.     Both  the  service  and  the  contractor  hoped  that  after  the  initial 
money  was  spent,    the  government  would  be  so  committed  to  the  program 
that  it  would  provide  successively  greater  increments  of  funding  rather 
than  cancel  it.     A  similar  "liars  contest"  was  conducted  regarding  the 
project  schedule.     If  the  government  wanted  the  equipment  by  a  certain 
date,    the  contractor  was  willing  to  commit  himself  to  provide  the  equip- 
ment by  that  date,    regardless  of  the  chances  of  doing  it,    again  often  at 
the  expense  of  test  and  evaluation,    which  was  the  final  activity  in  the     >. 
schedule. 
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B.      McNAMARA'S  POLICIES 

When  Robert  S.    McNamara  became  Secretary  of  Defense,   he  launched 
a  series  of  changes  in  the  government's  decision-making  processes  and 
in  weapon- system  contracting  procedures  which  profoundly  and  rapidly 
affected  the  defense  industry  environment.     He  began  by  centralizing 
much  of  the  authority  to  initiate  programs  in  the  Office  of  the  Secretary 
of  Defense.     To  avoid  duplication  of  procurement  in  the  services,   he 
utilized  the  concept  of  Systems  Analysis,    which  was  to  select  among 
competing  programs.     He  placed  government  budgeting  on  a  five-year 
basis  to  permit  appraisal  of  the  total  financial  impact  of  major  programs 
spanning  a  number  of  years.     He  required  contractors  to  plan  their 
activities  thoroughly  before  the  government  approved  the  program 
funding. 

To  dissuade  contractors  from  accepting  unrealistic  contracts  and  to 
encourage  efficient  performance  after  award,    McNamara  curtailed  the 
use  of  cost-plus-fixed-fee  contracts.      To  motivate  contractors  to  design 
weapon  systems  having  low  production  costs,    and  to  discourage  them 
from  bidding  low  on  weapon  system  contracts  in  the  hope  of  making  it  up 
in  the  subsequent  sole- source  production  contracts  he  authorized  the  use 
of  "Total  Package  Procurement"  contracts  in  which  the  development, 
test  and  evaluation,    production,    and  support  of  weapon  systems  were 
competitively  procured  in  a  single,    multi-year  contract. 
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The  Pentagon's  new  policy  was  bound  to  cause  great  difficulties  for 
the  defense  industry,    accustomed  as  it  was  to  the  old  way  of  doing 
business.     Its  problems  were  complicated  by  the  failure  of  the  Defense 
Department  to  adapt  completely  to  its  changed  role.     DOD  was  weak,    for 
example,    in  estimating  program  costs  and  in  predicting  how  much  test 
and  evaluation  was  necessary  to  reduce  uncertainty  in  weapon  system 
procurement  to  a  manageable  level.      Policies  established  at  high  levels 
were  not  understood,    implemented,    or  enforced  at  lower  levels.     The 
bureaucracy  was  doing  a  very  poor  job  of  defining  the  documentation 
necessary  to  implement  the  new  procedures. 

Because  the  contractors  were  accustomed  to  telling  their  customers 
what  they  wanted  to  hear  about  delivery  dates,    program  cost,    and  prod- 
uct performance,    they  found  it  almost  impossible  to  bring  themselves  to 
challenge  the  government's  preliminary  plans.     Yet  this  kind  of  confronta. 
tion  was  one  of  the  major  objectives  of  the  contract  definition  phase  of 
Total  Package  Procurement. 

Contractors,    would,    for  example,    consider  the  Pentagon's  requested 
delivery  dates  as  sacred,    and  tried  somehow  to  schedule  the  design, 
development,    test  and  evaluation,    and  production  into  the  available  time. 
They  tended  to  make  cost  and  technical  appraisals  by  automatically 
accepting  government  requirements  for  the  various  elements  of  the 
system  and  program  --   rather  than  by  estimating  each  element's  cost 
or  technical  performance  individually,    then  summarizing  them  to 
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obtain  an  independent  appraisal  of  total  system  performance  and  total 
program  cost. 

As  a  result,    the  weapon  system  procurement  plans  were  not  much 
more  realistic  than  those  formerly  submitted  by  contractors,    when  they 
simply  told  the  services  whatever  would  help  them  "sell"  the  programs 
further  up  the  line.     As  in  former  years,    when  schedule  and/or  cost 
crunches  arose,    one  of  the  first  areas  to  be  lopped  off  was  the  area  of 
test  and  evaluation  in  an  effort  to  get  the  hardware  delivered.      Paper 
studies  were  often  substituted  for  actual  test  and  evaluation  of  end 
product  items.     In  short,    contractors  often  signed  fixed-price,    total- 
package  contracts  at  prices  below  the  expected  cost,    containing  risks 
that  were  not  thoroughly  appraised,    and  for  which  they  lacked  the 
management  disciplines  necessary  to  perform  the  work  in  an  efficient 
manner. 

C.      GET  WELL  PROGRAMS 

Efforts  to  correct  problems  related  to  test  and  evaluation  (T&E) 
were  started  in  the  late  Sixties.     These  efforts  were  greatly  accelerated 
with  the  Report  of  the  Blue  Ribbon  Defense  Panel  (BRDP)  of  July  1970, 
and  various   T&E  measures  taken  during  calendar  1971  to  deal  with  the 
issues  raised  by  the  BRDP. 

1.       The  Blue  Ribbon  Defense  Panel's   Findings 

The  following  excerpt  is  quoted  from  the  BRDP  section  on 
"Operational  Testing  and  Evaluation,  "  pages  88-91  of  the  report  of 
1  July  1970. 

28  I       \ 


Everyone  seems  to  agree  that  Operational  Testing  and 
Evaluation  (OT&E)  is  very  important;  however,    there  are  significant 
differences  of  opinion  as  to  what  it  encompasses,    what  its  proper  objec- 
tives are,   and  what  organization  and  methods  are   necessary  to  accom- 
plish it  most  effectively. 

It  has  been  customary  to  think  of  OT&E  in  terms  of  physical 
testing  (under  various  designations  such  as  operational  suitability 
testing,    employment  testing,    service  testing,    or  field  experimentation). 
It  is  essential  to  recognize  that  the  primary  goal  of  OT&E  is  operational 
evaluation,    and  that  while  operational  testing  is  very  important  it  is  only 
one  method  of  evaluation.      To  be  effective,    OT&tE  must  be  a  total  pro- 
cess,  using  all  appropriate  methods  of  evaluation,    which  spans  the 
entire  cycle  of  a  system  from  initial  requirement  until  it  is  phased  out 
of  the  operational  forces.      If  OT&E  were  limited  to  physical  testing,    it 
would  lose  much  of  its  opportunity  to  contribute  to  decisions  on  whether 
to  produce  a  system,    and  would  seldom  be  able  even  to  influence  the 
system's  characteristics  and  capabilities  in  any  major  way. 

Much  OT&E  does,   however,    involve  physical  testing  and, 
therefor,    it  is  important  to  distinguish  between  "functional"  testing  and 
"operation"  testing. 

Functional  testing  (often  engineering  testing)  ((termed  "Develop- 
ment Testing"  in  this  Guide))  is  done  to  determine  how  well  various 
systems  and  materiel  meet  design  and  performance  contractual  specif- 
ications --in  other  words,    whether  they  meet  technical  requirements. 

By  and  large,    functional  testing  in  and  for  the  Department  of 
Defense  appears  to  be  well  understood  and  faithfully  executed.     Serious 
policy  deficiencies  are  not  apparent,    and  such  failures  in  functional 
testing  as  occur  can  be  primarily  attributed  to  lack  of  technical  com- 
petence,   oversight,'  or  procedural  breakdowns.      Functional  testing  is 
not  considered  to  be  a  major  problem  area. 

Operational  testing,    on  the  other  hand,    is  done  to  determine  to 
the  extent  possible  whether  such  systems  and  material  can  meet  opera- 
tional requirements.      It  must  provide  advance  knowledge  as  to  what 
their  capabilities  and  limitations  will  be  when  they  are  subjected  to  the 
stresses  of  the  environment  for  which  they  were  designed  (usually 
combat).     Operational  testing  must  take  into  account  the  interface  with 
other  systems  and  equipment,    tactics  and  techniques,    organizational 
arrangements,    and  the  human  skills  and  frailties  of  the  eventual  users. 
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There  has  been  an  increasing  desire,    particularly  at  OSD  level, 
to  use  data  from  OT&E  to  assist  in  the  decision-making  process.     Un- 
questionably,   it  would  be  extremely  useful  to  replace  or  support  critical 
assumptions  and  educated  guesses  with  quantitative  data  obtained  from 
realistic  and  relevant  operational  testing. 

Unfortunately,    it  has  been  almost  impossible  to  obtain  test 
results  which  are  directly  applicable  to  decisions  or  useful  for  analyses. 
Often  test  data  do  not  exist.     When  they  do,    they  frequently  are  derived 
from  tests  which  were  poorly  designed  or  conducted  under  insufficiently 
controlled  conditions  to  permit  valid  comparisons.     It  is  especially 
difficult  to  obtain  test  data  in  time  to  assist  in  decision-making.     Sig- 
nificant changes  are  essential  if  OT&E  is  to  realize  its  potential  for 
contributing  to  important  decisions,    particularly  where  the  tests  and 
the  decisions  must  cross  Service  lines. 

The  Blue  Ribbon  Defense  Panel  made  three  recommendations  on 
improvement  of  T&E.     While  subsequent  actions  do  not  constitute  rote 
implementation  of  the  letter  of  specific  recommendations,    they  do  con- 
stitute substantial  implementation  of  their  spirit.     BRDP  recommenda- 
tions called  for  establishment  of  a  "Defense  Test  Agency"  under  the 
control  of  an  Assistant  Secretary  of  Defense  for  Test  and  Evaluation. 
The  rationale  of  these  recommendations  was  (1)  to  make  sure  that 
adequate  testing  was  accomplished  by  an  agency  not  under  the  control 
of  the  people  who  were  attempting  to  "sell"  the  system  being  tested  and 
(2)  to  provide  an  unfiltered  channel  for  test-based  information  into  the 
major  program  decisions  made  by  the  Secretary  of  Defense  at  DSARC  I 
and  DSARC  II.     A  third  recommendation  called  for  a  separate  program 
category  for  T&E.     This  measure  was  designed  to  prevent  the  use  of 
funds  required  for  an  adequate  T&E  to  cover  development  cost  overruns. 
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2.       Mr.    Packard  and  the  DSARC 

One  of  the  first  things  that  Mr.    David  Packard  of  Hewlitt- 
Packard  realized  when  he  came  into  DOD  was  the  necessity  for  clarify- 
ing the  responsibilities  of  the  Secretary  of  Defense  and  his  office  vis-a- 
vis  those  of  the  Service  Secretaries  and  their  staffs.     He  quickly  decided 
that  there  was  a  need  for  a  very  simple  modus  operandi  by  which  the 
OSD  would  establish  policy  and  make  decisions  at  appropriate  points 
and  the  military  services  would  carry  them  out.     Once  having  done  its 
part,    the  OSD  would  step  back  out  of  the  way  of  the  performing  organiza- 
tion.     Furthermore,    the  various  OSD  offices,    such  as  that  of  the  Director 
of  Defense  Research  and  Engineering  and  the  Assistant  Secretaries  for 
Installations  and  Logistics,    Systems  Analysis,    and  Comptroller,    should 
lay  out  their  spheres  of  responsibility  and  authority,    subject,    of  course, 
to  Mr.   Packard's  approval. 

In  a  Deputy  Secretary  of  Defense  Memorandum  dated  28  May 
1970  entitled  "Policy  Guidance  on  Major  Weapon  System  Acquisition,  " 
Mr.    Packard  required  the  Services  to  develop  and  demonstrate  the 
operability  of  a  weapons  system  by  actual  test  prior  to  the  commitment 
of  funds  for  production  of  a  major  system.      Progress  in  meeting  this 
requirement  is  reviewed  by  the  Defense  Systems  Acquisition  Review 
Council  (DSARC)  at  discrete  stages  during  development.     DSARC 
determines  whether  the  system  is   ready  to  advance  to  the  next  stage, 
and  then  makes   recommendations  to  the  Secretary  of  Defense  concerning 
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funding  of  the  next  stage.      Generally  these  reviews  are  held  prior  to 
initiation  of  1)  engineering  development,    2)  prototyping,    and  3)  system 
production. 

When  the  council  considers  the  first  two  of  the  three  major 
decisions,    the  Director  of  Defense  Research  and  Engineering  chairs  it. 
For  the  third  decision,   he  serves  as  a  member,    while  the  Assistant 
Secretary  of  Defense  for  Installation  and  Logistics,    who  is  responsible 
for  production  and  development,   becomes  the  chairman.     At  this  point, 
the  Director  of  Defense  Research  and  Engineering  is  the  seller  and  the 
Assistant  Secretary  of  Defense  for  Installation  and  Logistics  is  the 
buyer. 

After  the  DSARC  finishes  its  deliberations,    the  Deputy  Sec- 
retary of  Defense  makes  a  decision  which  holds  unless  new  evidence 
arises  to  change  it.      For  example,   usually  the  program  goes  ahead 
smoothly  until  it  runs  into  some  kind  of  cost,    schedule,    or  system- 
performance  problem.      If  any  such  difficulty  causes  the  breaching  of 
established  thresholds  (i.  e.  ,    tolerances  on  critical  program  parameters 
agreed  upon  by  the  Secretary  of  Defense  and  the  responsible  Service 
Secretary),    the  program  comes  up  for  another  examination  by  the 
Secretary  of  Defense.      Barring  that  kind  of  situation,    the  programs 
are  run  by  the  Services  and  the  Secretary  of  Defense  stays  out  of  them  -- 
except  to  keep  informed  on  their  status  through  regular  program 
reviews  and  Selected  Acquisition  Reports. 
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In  his  memo  of  11  February  1971,    "Conduct  of  Operational 

Test  and  Evaluation,  "  Secretary  Packard  wrote: 

Although  each  Service  now  has  a  somewhat  different  way  of 
organizing  for  operational  test  and  evaluation,    it  is  apparent  to  me 
that  this  function  can  best  be  performed  by  an  agency  which  is  separate 
and  distinct  from  the  developing  command  and  which  reports  the  results 
of  its  test  and  evaluation  efforts  directly  to  the  Chief  of  the  Service. 
Moreover,    within  the  Service  headquarters  staff,    there  needs  to  be  an 
office  with  a  clear  OT&E  identification  to  provide  staff  assistance 
directly  to  the  Service  Chief  and  to  provide  a  headquarters  focal  point 
for  the  independent  OT&E  field  agency.      Thus,   at  the  completion  of 
Operational  Service  Testing  in  the  Army,    OPEVAL,  in  the  Navy,    and 
Cat  III  testing  in  the  Air  Force,    I  would  expect  that  the  respective 
Service  Chiefs  would  have  a  clear  picture  of  the  operational  suitability 
of  a  weapon  system  for  Service  use,    to  include  its  principal  deficiencies 
and  limitations  and  the  corrective  actions   required  prior  to  full-scale 
introduction  into  the  force.     Accordingly,    each  Service  is  requested  to 
restructure  its  organization  for  OT&E  along  the  lines  specified  above. 

As  a  second  step,    I  am  establishing  a  Deputy  Director  for  Test 
and  Evaluation  within  ODDR&E  with  across  the  board  responsibilities  for 
OSD  in  test  and  evaluation  matters.     This  office  will  review  and  approve 
test  and  evaluation  plans  prepared  by  the  Services  and  will  provide  an 
assessment  of  results  obtained. 

In  his  memo  of  21  April  1971,    Secretary  Packard  prescribed 

measures  to  ensure  the  flow  of  T&E  information  into  major  program 

decisions. 

The  DCP  format  focuses  attention  on  relevant  technical  and 
operational  information  through  the  identification  of  achievement  mile- 
stones.     The  function  of  the  test  and  evaluation  milestones  is  to  insure 
the  availability  of  critical  technical  and  operational  data  prior  to  each 
major  Defense  System  Acquisition  Review  Council  action. 

The  newly-established  Deputy  Director,    Defense  Research 
and  Engineering  (Test  and  Evaluation)  will  receive  from  the  Military 
Departments  for  review  and  comment,    during  the  drafting  of  the  initial 
DCP,   a  brief  (1-4  pages)  statement  of  the  critical  questions  or  issues 
which  the  development  tests  will  address.     The  coordinated  DCP  for 
DSARC  Milestone  I  (Program  Decision)  will  contain  a  condensed 
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version  of  the  critical  questions  or  issues  as  well  as  the  proposed 
schedule  of  the  major  development  test  milestones  for  addressing  them. 

The  DCP  for  DSARC  Milestone  II  (Ratification  Decision)  will 
refine  the  critical  questions  or  issues  associated  with  development  test 
and  evaluation  and  will  extend  the  process  to  include  the  critical  ques- 
tions or  issues  to  be  addressed  during  initial  OT&E  and  the  test  sched- 
ules therefor.     At  such  time  as  the  OT&E  test  plan  is  available,    the 
Deputy  Director  will  review  it  to  insure  that  the  critical  questions  or 
issues  are  adequately  covered. 

Prior  to  the  DSARC  Milestone  III  (Production  Decision)  the 
Military  Departments  will  provide  the  Director  of  Defense  Research 
and  Engineering  with  an  assessment  of  the  test  results  in  terms  of 
response  to  the  initial  questions  or  issues  previously  identified.      The 
Deputy  Director,    Test  and  Evaluation,    will  review  this  assessment 
and  provide  an  independent  recommendation  to  the  DSARC  Milestone 
III  meeting. 

3.       Department  of  Defense  Directive  5000.  1 

Department  of  Defense  Directive     (DODD)  5000.  1,    Acquisition 
of  Major  Defense  Systems,    the  cornerstone  of  the  Navy's  comprehensive 
new  acquisition  policy,    was  the  culmination  of  a  concerted  effort  to 
identify  the  underlying  causes  of  unsuccessful  acquisition  experiences 
and  to  prompt  whatever  improvements  were  deemed  necessary.     While 
DODD  5000.  1  addresses  major  weapon  system  programs,    the  manage- 
ment principles  on  which  it  is  based  are  applicable  to  all  programs. 
The  directive  formally  establishes  the  DSARC.     It  takes  a  strong 
position  on  the  decentralization  of  high-level  controls  in  regard  to  the 
day-to-day  aspects  of  acquisition,    and  it  establishes  the  concept  of  a 
powerful,   less  encumbered  program  manager.      Finally,    DODD  5000.  1 
establishes  a  number  of  considerations  for  major  weapon  system 
acquisition. 
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In  the  area  of  test  and  evaluation,    paragraphs  III.  C.  5  and  6 
apply  directly  and  are  as  follows: 

5.  Technical  uncertainty  shall  be  continually  assessed. 
Progressive  commitments  of  resources  which  incur  program  risk  will 
be  made  only  when  confidence  in  program  outcome  is  sufficiently  high 
to  warrant  going  ahead.     Models,    mock-ups  and  system  hardware  will 
be  used  to  the  greatest  possible  extent  to  increase  confidence  level. 

6.  Test  and  evaluation  shall  commence  as  early  as  possible. 
A  determination  of  operational  suitability,    including  logistic  support 
requirements,    will  be  made  prior  to  large-scale  production  commit- 
ments,   making  use  of  the  most  realistic  test  environment  possible  and 
the  best  representation  of  the  future  operational  system  available. 
The  results  of  this  operational  testing  will  be  evaluated  and  presented 
to  the  DSARC  at  the  time  of  the  production  decision. 

4.       More  Guidance  From  Mr.    Packard 

Secretary  Packard's  memo  of  3  August  1971,    "Test  and 

Evaluation  in  Systems  Acquisition  Process,  "  amplified  the  guidance 

of  his   11   February  and  21  April  memos  in  part  as  follows: 

The  objective  of  the  overall  operational  test  and  evaluation 
effort  for  any  program  is  to  aid  in  providing  at  major  decision  points 
in  the  acquisition  and  development  process  the  best  information  possible 
at  that  point  in  time  as  to:     the  military  utility  of  the  prospective  system; 
its  expected  operational  effectiveness,    operational  suitability  (including 
reliability,    logistic,    and  training  requirements);  need  for  modification; 
and  the  organization,    doctrine  and  tactics  for  system  deployment. 
For  programs  intended  for  acquisition,    phases  of  operational  test  and 
evaluation  must  be  successfully  executed  in  a  timely  manner  to  provide 
needed  information  as   required. 

New  acquisition  programs  requiring  DSARC  processing,    or 
those  which  are  currently  in  their  early  stages,    will  be  so  executed 
that  an  initial  phase  of  operational  test  and  evaluation  will  be  accomp- 
lished prior  to  the  major  production  decision  to  assist  in  estimating 
system  operational  effectiveness  and  suitability  before  that  decision. 

a.      This  initial  operational  test  and  evaluation  will  be 
accomplished  with  operational  personnel  in  as   realistic  an  operating 
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environment  as  possible  and,    where  practical,    will  use  pilot  or  early- 
production  items. 

The  accomplishment  of  initial  operational  test  and 
evaluation  as  a  prerequisite  for  production  decision  does  not  obviate 
the  necessity  for  completing  operational  test  and  evaluation  when  the 
production  articles  are  introduced  into  the  operating  forces. 

5.       From  the  Deputy  DDR&E(T&E) 

In  a  memorandum  for  the  Assistant  Directors,    the  Deputy 

DDR&E(T&E)  reviewed  the  three  DepSecDef  memos  cited  above  and 

outlined  the  following  functions  for  his  office: 

In  summary  then,    the  Deputy  Director  Test  &  Evaluation 
(DDT&E)  (with  the  assistance  of  his  staff)  is  to  accomplish  the  following: 

1.  Have  across-the-board  responsibilities  for  test  and 
evaluation  in  the  Department  of   Defense.     Review  test  and  evaluation 
policies  and  procedures  of  the  DOD  and  the  Military  Departments  and 
recommend  changes  as  appropriate. 

2.  Monitor  closely  test  and  evaluation  programs  conducted 
by  the  Services  for  DSARC  programs  and  such  other  programs  as  he 
believes  necessary  throughout  the  entire  testing  cycle.     Report  to  the 
DSARC  and  directly  to  DepSecDef  at  DSARC  Milestones  I  and  II  his 
assessment  as  to  the  adequacy  of  the  list  of  critical  issues  and  prob- 
lems to  be  attacked  by  test  and  evaluation  and  the  schedule  of  test 
milestones,    and  report  at  Milestone  III  to  the  DSARC  and  to  the  Dep- 
SecDef his  independent  recommendation. 

3.  Insure  timely  OT&E  to  meet  the  intent  of  DepSecDef's 
direction  described  in  2a  above. 

4.  Request  Service  test  plans  and  test  results  as  may  be 
required  to  accomplish  (1),    (2),   and  (3)  above  as  early  as  such  plans 
are  developed  by  the  Services  and  needed  by  DDT&E. 

5.  Initiate  and  coordinate  appropriate  joint  testing. 

6.  Oversee  the  evaluation  of  foreign  systems  for  possible 
DOD  use. 
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7.  Administer  for  OSD  its  responsibility  for  the  national 
and  major  Service  ranges. 

8.  Carry  out  such  other  T&E  related  activities  as  the  SecDef, 
DepSecDef,    DDR&E,    or  other  DSARC  member  may  direct. 

6.       More  from  The  Deputy 

LTGEN  A.    D.    Starbird,    the  Deputy  Director  Defense  Research 

and  Engineering  for  Test  and  Evaluation,    in  an  address  to  the  7th  Naval 

Aviation  Test  and  Evaluation  Conference  at  NATC  Patuxent  River,    Md.  , 

on  24  August  1971  commented  on  the  evolution  of  test  and  evaluation 

concepts  and  emerging  policies  as  follows: 

a.  As  we  all  know  the  Military  Departments  and  the  DOD  have 
increasingly  come  into  criticism  in  almost  all  phases  of  their  activities. 
Unfortunately,    test  and  evaluation  has  not  escaped.     In  fact,    on  many 
occasions  it  has  come  in  for  severe  criticism.     Each  time  a  system  in 
rather  advanced  stage  of  acquisition  runs  into  trouble,    there  are  critics 
who  center  on  inadequate  test  and  evaluation  as  a  major  culprit. 

b.  It  is  interesting  to  me  that  the  criticism  of  test  and  evalua- 
tion has  come  from  many  different  quarters.     On  occasion,    the  press 
and  the  public  have  been  critical.      Congess  in  general  has  a  sincere 
interest  in  seeing  that  we  do  adequate  test  and  evaluation  and  their 
criticisms  have  been  fairly  frequent.      The  GAO  has  been  critical, 
much  more  so  recently  than  in  earlier  years.      The  President's 
Scientific  Advisory  Committee  was  asked  to  study  out  T&E  and  they 
emerged  with  certain  criticisms.     Many  of  you  are  probably  familiar 
with  the  fact  that  the  President's  Blue  Ribbon  Defense  Panel,    which 
reported  about  a  year  ago,    was  quite  critical  of  test  and  evaluation 

and  particularly  the  quality  of  the  operational  test  and  evaluation 
accomplished  by  the  Departments. 

c.  Some  of  the  critics  have  called  for  rather  radical  changes 
in  how  we  do  our  T&E  business.      The  Blue  Ribbon  Defense  Panel,    for 
example,    recommended:     appointment  of  an  Assistant  Secretary  of 
Defense  for  Test  and  Evaluation  to  have  a  strong  hand  in  approving 
Service  test  plans;  and  an  OSD-level  Test  and  Evaluation  Agency  to 
carry  out  the  more  critical  DOD  program  tests. 
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We  all  realize  that  many  of  the  criticisms  leveled  at  our  T&E 
are  just  not  justified.     "We  in  the  DOD  have  done  in  the  past,    and  are 
doing  now,    much  excellent  testing.      The  results  in  the  effectiveness  of 
many  of  our  complex  systems  show  that  our  test  and  evaluation  have 
been  good.     We  devote  a  tremendous  amount  of  effort  of  the  best  people 
and  at  high  cost  to  test  and  evaluation.     On  balance,    I  believe  that  the 
Services  are  on  the  par  with  any  other  large  industry,    and  perhaps 
ahead  of  most. 

But  we  have  had  faults  and  some  of  those  persist.      For  these 
we  are  rightfully  criticized.      The  faults  in  general  fall  in  two  categories. 

First  is  the  availability  of  test  results.      Frequently  the  testing 
and  its  evaluation  are  completed  long  after  critical  decisions  and  major 
expenditure  of  funds  have  been  made.     We  are  well  into  production 
before  we  realize  from  tests  that  we  will  not  reach  the  reliability  goals 
specified.     We  are  well  into  deployment  on  occasion  before  we  know 
that  we  cannot  maintain  adequately.      The  first  valid  criticism  then  is 
that  much  of  our  testing  is  not  timely. 

Second,    our  key  testing  to  determine  operational  effectiveness 
and  suitability  is  often  not  accomplished  until  we  are  irrevocably 
committed  at  large  dollar  cost  to  some  particular  design.     We  do  not 
accomplish  early  enough  adequate  operational  test  and  evaluation. 
There  are  many  definitions  of  operational  testing  but  I  think  we  all 
know  what  it  means.     It  is  testing  of  a  product  reasonably  representa- 
tive of  the  final  design,    by  operational  units,    in  the  type  of  environment 
the  system  will  encounter  when  deployed,    so  as  to  establish  its  military 
worth  and  operational  suitability.      Included  in  that  matter  of  operational 
suitability  is  whether  its   reliability  requirements  can  be  met  and  with- 
out undue  maintenance. 

Secretary  Laird  and  Secretary  Packard  have  both  been 
extremely  interested  in  our  test  and  evaluation  from  the  time  they  took 
office  with  the  Department  of  Defense.     I  remember  two  long  sessions 
within  the  first  week  after  they  were  sworn  in  when  they  met  during 
most  of  one  evening  on  a  system  for  which  I  then  had  responsibility. 
Both  asked  many,    many  questions  dealing  with  test  organizations, 
test  results  and  test  plans.      Both  for  many  months  have  discussed 
with  the  Service  Secretaries  and  the  Military  Chiefs  what  steps  were 
necessary  to  accomplish  improvements.      Beginning  early  this  calendar 
year,    they  issued  a  series  of  three  instructions  indicating  what  changes 
they  desired  to  be  made.      The  latest  is  the  instruction  by  Secretary 
Packard  of  3  August  to  which  I  earlier  referred.      Reading  these  col- 
lectively you  can  see  that  their  objectives  are  really  four  in  number. 
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First,    and  I  believe  this  their  primary  objective,    they  desire 
to  improve  rather  than  revolutionize  our  test  and  evaluation  approach. 
They  want  to  build  on  the  good  that  exists  rather  than  to  start  again 
from  scratch. 

They  want  to  avoid  "adding  testing  just  for  testing's  sake.  " 
They  insist  that  everyone  know  the  objective  of  their  testing  before 
embarking  on  their  test  program. 

They  insist  that  the  testing  be  timely,    --  that  it  be  adequate 
and  scheduled  for  evaluation  in  time  to  contribute  to  the  major  decisions 
they  must  make.      (I  should  point  out  that,    for  all  important  systems 
being  acquired  by  the  Services,    they  must  approve  moving  a  program 
from  concept  definition  into  advanced  development,    the  subsequent 
movement  into  engineering  development,    and  the  third  and  more 
important  decision  to  move  into  production.  ) 

They  insist  too  that  the  testing  be  realistic,    that  it  be  in 
sufficient  detail  and  adequately  evaluated  to  give  realistic  answers. 

7.       T&E  and  The  Congress 

Congress,    recognizing  the  importance  of  test  and  evaluation 

in  the  procurement  of  weapons  systems,    enacted  in  Public  Law  92-156 

of  November   17,    1971  a  Section  506.     A  partial  quote  of  that  Section.as 

applies  to  test  and  evaluation  follows: 

"Sec.    506.      (a)     Beginning  with  the  calendar  year   1972,    the  Secretary 
of  Defense  shall  submit  to  the  Congress  each  calendar  year,    at  the 
same  time  the  President  submits  the  Budget  ...    a  written  report 
regarding  development  and  procurement  schedules  for  each  weapon 
system  for  which  fund  authorization  is  required  .  .  .      Beginning  with  the 
calendar  year   1973,    there  shall  be  included  in  the  report  data  on 
operational  testing  and  evaluation  for  each  such  weapon  system  for 
which  funds  for  procurement  are  requested  (other  than  funds   requested 
only  for  the  procurement  of  units  for  operational  testing  and  evaluation 
and/or  long  lead-time  items)  .  .  . 

"(b)    A  supplemental  report  shall  be  submitted  to  the 
Congress  by  the  Secretary  of  Defense  not  less  than  thirty  nor  more 
than  sixty  days  before  the  awarding  of  any  contract  or  the  exercising 
of  any  option  in  a  contract  for  the  procurement  of  any  such  weapon 
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system  (other  than  procurement  of  units  for  operational  testing  and 
evaluation  and/or  long  lead-time  items)  .  .  . 

"(c)    Any  report  required  .  .  .    shall  include  detailed  and 
summarized  information  with  respect  to  each  weapon  system  covered 
by  such  report,    and  shall  specifically  include,    but  shall  not  be  limited 
to  --  ... 

"(3)  .  .  .    the  results  of  all  operational  testing  and  evaluation 
up  to  the  time  of  the  submission  of  the  report,    or,    if  operational  testing 
and  evaluation  has  not  been  conducted,    a  statement  of  the  reasons  there- 
for and  the  results  of  such  other  testing  and  evaluation  as  has  been 
conducted.  " 


D.      IMPLEMENTATION  DIRECTIVES  AND  INSTRUCTIONS 
1.       SECNAVINST  5000.  1 

To  implement  DODD  5000.  1,    the  Navy  issued  SECNAVINST 
5000.  1  and  other  flowdown  directives  which  will  be  discussed  later. 
Paragraph  6e  of  the  basic  instruction  assigns   COMOPTEVFOR 
(Commander,    Operational  Test  and  Evaluation  Force)  responsibility 
as  an  independent  test  agency  for  weapon  system  acquisition  required 
T&E.     It  is  interesting  to  note  that  enclosure  (2)  of  the  basic  instruction 
canceled  28  previous  instructions,    of  which  three  had  T&E  in  their 
titles. 

Enclosure  (3)  to  the  basic  instruction  entitled  "Policy, 
Relationships  and  Responsibilities"  contains  the  real  meat  of  system 
acquisition  direction  within  the  Navy.      For  designated  projects   (Major 
Defense  Systems  such  as  those  in  Selected  Acquisition  Reports),    the 
mode  of  operation  for  Project  Managers  is  prescribed.      The  Project 
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Manager  is  directed  to  establish  coordination  with  the  DRDT&E 
(Director,    Research,    Development  Test  and  Evaluation)  to  ensure 
adequate  operational  test  planning  and  operational  test  and  evaluation 
accomplishment.     Section  II  of  enclosure  (3)  indicates  that  test  and 
evaluation  is  to  be  a  consideration  in  RDT&E  conceptual  effort,    full- 
scale  development,    and  preproduction.     Paragraph  II.  C.  2.  b.    states 
that  programs  will  not  normally  be  proposed  to  the  DSARC  for  SECDEF 
approval  for  production  prior  to  completion  of  requisite  operational 
test  and  evaluation.     Deviations  to  prescribed  T&E  can  be  made  only 
by  or  with  the  concurrence  of  the  SECNAV. 

Section  II,    "Program  Considerations,  "  in  paragraph  D 
further  spells  out  T&E  policy  as  follows: 

1.       The  wide  variety  of  naval  weapons  dictates  varying 
approaches  to  the  conduct  of  test  and  evaluation;  such  effort  shall  be 
tailored  to  the  needs  and  characterizations  of  each  individual  acquisi- 
tion --  prime  consideration  being  given  to  adequate  operationally 
oriented  testing.      Normally,    the  following  general  sequence  of  events 
should  prevail;   (1)  laboratory/contractor  preliminary  test  and  evalua- 
tion of  breadboard  of  demonstration  hardware  during  the  conceptual 
effort,    (2)  contractor/development  activity  test  and  evaluation  of  sub- 
systems and/or  full-scale  demonstrator  hardware  during  full-scale 
development,    (3)  technical  test  and  evaluation  conducted  by  the  con- 
tractor with  Navy/Marine  Corps  participation  during  pre-production/ 
production,    (4)  IOT&E  (Initial  Operational  Test  and  Evaluation)  by  or 
with  the  active  participation  of  Navy  operational  forces  prior  to  the 
major  production  decision,    (5)  Navy/Marine  Corps  OPEVAL  (Opera- 
tional Test  and  Evaluation)  prior  to  approval  for  service  use  and 
inventory  acceptance  (except  for  ships),    (6)  Navy /Marine  Corps 
follow-on  test  and  evaluation,    and  (7)  the  conduct  of  tests  and  evalua- 
tions by  the  Board  of  Inspection  and  Survey  (BIS)  and  recommendation 
to  the   Chief  of  Naval  Operations  for  service  acceptability.      Test  and 
evaluation  effort  shall  be  effectively  correlated  with  previously  out- 
lined requirements  concerning  approval  of  material  for  service  use. 
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Specifically,    the  procedural  aspects  of  requirements  determination, 
research  and  development,    manufacture  of  service  test  model(s),    tech- 
nical evaluation,    initial  operational  test  and  evaluation,    full  operational 
evaluation,    and  approval  for  service  use  shall  be  correlated. 

2.  New  acquisition  includes  conversions,    major  modifications 
and  modernizations.     Adequate  test  and  evaluation  of  these  is  also 
required  

3.  For  new  acquisitions  .  .  .    the  principles  of  early  operational 
test  and  evaluation  before  production  decision,    participation  by  the 
Marine  Corps  or  COMOPTEVFOR  shall  apply.      The  purpose  of  IOT&E 
and  other  tests  is  to  provide  assessments  and  recommendations  by  an 
activity  independent  from  the  development  activity  concerning  the  future 
operational  suitability  of  systems  under  development. 

4.  The  objective  of  the  overall  operational  test  and  evaluation 
effort  for  any  program  is  to  aid  in  providing,    at  major  decision  points  in 
the  development  and  acquisition  process,    the  best  information  possible 
at  that  point  in  time  as  to:     the  military  utility  of  the  prospective  system; 
its  expected  operational  effectiveness,    operational  suitability  (including 
reliability,    maintainability,    simplicity,    logistic,    and  training  require- 
ments); need  for  modifications;  and  the  organization,    doctrine  and 
tactics  for  system  deployment  .  .  . 

2.       Congressional  Concern 

Concurrent  with  the  issuance  of  DODD  5000.  1  and  the  Navy's 
implementing  instruction  SECNAV  5000.  1,    there  was  increasing  Con- 
gressional concern  over  testing  of  major  acquisitions.      Because  of  the 
major  problems  encountered  with  weapons  programs  as  a  result  of 
inadequate  testing,    Congressional  committees  increasingly  qualified 
the  authorization  of  resources  assigned  to  programs  until  comprehen- 
sive testing  had  been  completed.      Following  are  some  examples  from 
the  fiscal  year   1971  and   1972  authorizing  appropriations  to  illustrate 
this  increased  interest  by  the  Congress.      The  restraints  applied  most 
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often  occur  during  requests  by  the  services  to  place  a  major  weapon 

in  full-scale  production. 

In  Public  Law  91-441,   dated  October  7,    1970,    the  Congress 

stated  that: 

Of  the  total  amount  authorized  to  be  appropriated  by  this  Act  for  the 
procurement  of  the  F-lll  aircraft,    $283,  000,  000,    of  such  amount  may 
not  be  obligated  or  expended  for  the  procurement  of  such  aircraft  until 
and  unless  the  Secretary  of  Defense  has  (1)  determined  that  the  F-lll 
aircraft  has  been  subjected  to  and  successfully  completed  a  comprehen- 
sive structural  integrity  test  program  .  .  . 

In  House  Report  92-232  on  authorization  appropriations  for 

fiscal  year  1972,   SRAM  and  MAVERICK  were  discussed. 

The  AGM-69A  SRAM  is  an  air  launched  air-to-surface  missile  for 
planned  use  on  the  B-52G/H  and  FB-111  aircraft.      The  SRAM  missile 
is  equipped  with  a  nuclear  warhead  designed  to  attack  targets  defended 
by  sophisticated  defense  systems.      The  fiscal  year  1972  program  pro- 
vides for  proceeding  to  full-scale  production  considering  completion  of 
testing. 

MAVERICK  is  an  air-to-surface  Air  Force  missile,    electro-optically 
guided  for  use  against  small  hard  targets  such  as  tanks  and  bunkers. 
Last  year  the  Congress,    on  the  recommendation  of  the  Committee  on 
Armed  Services,    denied  requested  procurement  funds  for  MAVERICK 
and  directed  that  the  program  be  continued  in  research  and  development 
to  avoid  concurrency  and  to  allow  more  reliable  development  and  test 
results  prior  to  initiating  procurement. 

3.       T&E  and  the  GAO 


At  the  request  of  the  Honorable  Carl  Albert,    Speaker  of  the 
House  of  Representatives,    the  General  Accounting  Office  (GAO)  made 

a  review  of  the  importance  of  testing  and  evaluation  by  the  Department 

2 
of  Defense  in  the  acquisition  process  for  major  weapon  systems. 


2 

The  Importance  of  Testing  and  Evaluation  in  the  Acquisition  Process 

for  Major  Weapons  Systems,    The  Comptroller  General  of  the  United 
States,    7  August  1972. 
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Their  report  indicates  that  the  benefit  of  testing  is  accomplished 
through  properly  assessing  risks  and  delivering  test  results  to  the 
decision-maker  at  key  points  in  the  acquisition  cycle  when  final  de- 
cisions must  be  made.      Testing  at  decisive  stages  of  development  was 
found  to  show  where  problems  exist  and  to  help  military  managers 
make  sound  decisions  affecting  future  production  and  purchase  of 
weapons  than  would  otherwise  be  possible. 

The  GAO  reviewed  13  weapon  systems  with  estimated  total 
costs  of  more  than  $46  billion.      They  included  such  weapons  as  the 
Army's  improved  HAWK  missile,    the  Navy's  DE-1052  (destroyer 
escort),    and  the  Air  Force's   F-15  aircraft.      On  the  basis  of  its  ob- 
servations of  the  pattern  of  testing  performance,    the  GAO  concluded 
that: 

--Practices  used  to  establish  objectives  for  testing  generally  were 
adequate. 

--Most  weapon  systems  did  not  have  adequate  plans  for  conducting 
tests. 

--Testing  and  evaluation  for  most  weapon  systems  was  not  accomplished 
in  a  timely  manner. 

--Most  test  reports  were  adequate,   but  their  value  was  diminished  due 
to  inadequate  test  planning  and  actual  testing.     Some  reporting  im- 
provements could  be  made. 

--Complete  and  valid  test  and  evaluation  data  was  not  available  prior 
to  those  times  in  the  acquisition  cycle  when  decisions  had  to  be 
made. 

Based  on  DOD's  listing  of  five  phases  in  the  acquisition 

process  of  weapon  systems   (1     concept  formulation,     2     validation  and 
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ratification,      3     development,     4    production,    and    5    deployment)  the 

GAO  formulated  a  model  (Appendix  A)  to  depict  the  role  of  testing  in 

the  acquisition  cycle.      The  purpose  of  GAO's  model  was  to  reinforce 

and  emphasize  certain  ideal  concepts  of  testing  and  evaluation  in  the 

acquisition  process,   as  follows: 

--Testing  and  evaluation  is  an  important  ingredient  throughout  the 
acquisition  process. 

--The  sequential  nature  of  engineering;  acceptance;  and,    to  some 
degree,    operational  suitability  testing  and  evaluation. 

--The  responsibility  for  the  success  of  testing  and  evaluation  through- 
out the  acquisition  process  lies  with  the  developer,    the  user,    and 
the  contractor  in  different  degrees  and  at  different  times. 

--To  provide  a  means  for  DOD  and  weapon  systems  program  managers 
to  evaluate  the  testing  and  evaluation  process  in  their  respective 
programs. 

4.       Laird's  Viewpoint 

In  his  final  report  to  the  House  Armed  Service  Committee  on 
January  8,    1973,   Secretary  Melvin  Laird  indicated  that  with  the  help  of 
David  Packard,    many  changes  had  been  made  in  the  weapons  system 
acquisition  process  but  that  it  would  take  some  years  before  the  im- 
provements would  be  fully  validated.     He  indicated  that  bankrupt  prac- 
tices such  as  total  package  procurement  and  an  indiscriminate  use  of 
concurrency  between  development  and  production  had  been  eliminated. 
His  substitutes  included  "test  before  you  fly"  and  "fly  before  you  buy" 
procedures,    and  the  widespread  use  of  contract  milestones  and 
prototyping. 
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5.       DODD  5000.3 

In  January  1973,    the  several  earlier  T&E  memoranda  were 

superseded  by  DOD  Directive  5000.  3  entitled  "Test  and  Evaluation.  " 

It  was  in  somewhat  more  detail  and  tightened  to  a  degree  the  earlier 

memoranda  instructions: 

--It  stated  certain  principles  that  should  apply  to  all  acquisition  pro- 
grams.     T&E  would  start  as  early  as  possible  and  be  conducted  in 
phases  so  as  to  eliminate  risks  early.     Acquisition  schedules  must 
be  keyed  to  accomplishing  T&E  milestones  before  major  added 
resource  commitments  were  made.      Prior  to  contracting  for  major 
production,    there  would  be  completed:     sufficient  development  test- 
ing (DT&E)  to  insure  that  all  design  problems  were  identified  and 
solutions  were  in  hand;  and  sufficient  operational  testing  (OT&E)  to 
provide  a  valid  estimate  of  operational  effectiveness  and  suitability. 

--In  each  Service,    OT&E  would  be  the  responsibility  of  a  field  agency 
independent  of  the  developing  command.      Typical  military  operating/ 
maintaining  personnel  and  units  were  to  carry  out  the  test,    --  not 
highly  trained  development  technicians.     Such  operational  testing 
was  to  be  conducted  in  a  simulated  operational  environment  where 
the  stresses  of  real  combat  operations  (such  as  fatigue  and  counter- 
measures)  were  applied  to  the  degree  possible. 

--Operational  tests  preferably  were  to  be  separate  from  development 
tests.     However,    an  early  phase  of  operational  testing  could  be 


46 


combined  with  a  development  test  providing  the  combined  test  was 
participated  in  and  accepted  by  both  the  developer  and  the  independent 
operational  test  agency  as  satisfying  their  individual  requirements. 
Pilot  production  items  were  to  be  used  wherever  possible  for  the 
development  and  operational  testing  completed  before  commitment 
to  major  production.     However,    final  prototypes  could  be  employed 
where  these  prototypes  were  truly  like  the  expected  production  items. 

--For  T&E  of  ships  of  a  major  class  it  was  recognized  that  because 
of  the  long  design,    engineering,    and  construction  period,    completion 
of  T&E  prior  to  the  decision  to  proceed  with  follow  ships  would  not 
in  all  likelihood  be  possible.      "In  lieu  thereof,    successive  phases  of 
DT&E  and  OT&E  will  be  accomplished  as  early  as  practicable  at  test 
installations   ...    to  minimize  the  need  for  modification  to  follow 
ships.  "    Use  of  prototypes  to  achieve  T&E  prior  to  the  first  major 
production  decision  on  follow  ships  is  advocated. 

--The  directive  included  an  important  new  provision  with  respect  to 
waiving  satisfactory  completion  of  required  T&E.      For  each  major 
acquisition  program,    the  describing  Development  Concept  Paper 
(DCP)  sets  forth  such  required  T&E  and  only  the  Secretary  of 
Defense  can  waive  its  accomplishment. 
6.       OPNAVIN5T  3660.  8 

OPNAV  INSTRUCTION  3660.8  of  22   January  1973  established 

guidance  and  policy  for  T&E  of  Navy  systems  and  equipments  in 
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accordance  with  the  requirements  of  SECNAVINST  5000.  1  and  DODD 
5000.  3.      The  instruction  reemphasizes  that  Navy  weapon  system  ac- 
quisitions shall  provide  for  T&E  programs  to  (1)  permit  a  clear  deter- 
mination of  service  acceptability  and  suitability  by  competent  authority; 
(2)  support  the  development  activities  in  evolving  a  design  which  meets 
specifications  and  service  requirements;  (3)  support  the  Secretary  of 
Defense  review  and  decision  process. 

It  recognized  that  conventional  ship  programs  may  be  large, 
complex  and  prolonged  evolutions.      Consequently,    development  T&E 
required  on  a  specified  subsystem  or  groups  of  subsystems  may  be 
conducted  on  board  other  class  ships  or  at  land-based  test  sites.     In 
the  case  of  a  new  hull  or  propulsion  designs  prototypes,    or  land-based 
test  sites,    construction  is  normally  undertaken  to  verify  laboratory 
DT&E. 

While  previous  instructions  and  directives  addressed  T&E  in 
general  terms,    3960.  8  defines  T&E  specifically  and  discusses  T&E  of 
weapons   systems  in  the  following  areas:     developmental  testing,    opera- 
tional testing,    and  acceptance  trials.     The  following  pertinent  definitions 
are  made: 

TEST:    Any  program  or  procedure  which  is  designed  to  obtain,    verify, 
or  provide  data  for  the  evaluation  of;   research  and  development;  pro- 
gress in  accomplishing  development  objectives;  or  performance  and 
operational  capability  of  system,    subsystems,    components,    and  equip- 
ment items. 
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DEVELOPMENTAL  TEST  AND  EVALUATION  (DT&E):     Test  and 
evaluation  performed  by  or  for  the  developing  agency  which  emphasizes 
the  technological  and  engineering  aspects  of  the  system,    subsystem,  or 
equipment  items.      Normally  carried  out  under  strictly  controlled 
conditions. 

OPERATIONAL  TEST  AND  EVALUATION  (OT&E):     Tests  and  evalua- 
tions participated  in  or  performed  by  operational  personnel  focusing  on 
operational  effectiveness  and  suitability  (including  reliability,    compat- 
ibility,   interoperability,    maintainability,    and  supportability).     It  also 
includes  the  development  of  optimum  operational  tactics  for  systems 
and  equipment  being  developed  for  service  use. 

OT&E  is  further  broken  down  into  the  following  areas: 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (IOT&E):     That 
T&E  accomplished  prior  to  the  DSARC  Milestone  III  or  comparable 
CNO  or  CHNAVMAT  first  major  production  decision  point  to  permit 
assessment  of  the  operational  effectiveness  and  suitability  of  a  weapon 
system. 

FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION  (FOT&E): 
The  continuing  operational  test  and  evaluation  of  a  weapon  system  con- 
ducted in  an  operational  environment  by  operational  personnel  using 
production  systems  for  the  purpose  of  verifying  system  performance; 
validating  correction  of  deficiencies  previously  identified;  and  refining 
tactical  employment  doctrine  and  requirements  for  personnel  and 
training.      FOT&E  may  be  initiated  using  pilot  or  pre-production  sys- 
tems which  most  closely  resemble  the  production  units  until  the  latter 
items  are  available. 

A  comprehensive  set  of  definitions  of  terms  and  phrases 

commonly  used  in  T&E,    checklists  for  T&E  in  support  of  milestones 

I  through  III  of  the  DSARC  process,    and  a  diagram  of  the  acquisition 

process  are  provided  as  enclosures.      T&E  responsibilities  and 

relationships  which  have  evolved  are  also  discussed  which  will  be  the 

subject  of  the  following  section  of  this  thesis. 
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7.       NAVMAT  NOTICE  3960 

NAVMAT  NOTICE  3960  of  1   February  1973,    "Test  and  Evalua- 
tion Policy  for  Major  Surface  Combatant  Ships  Project,  "  assigned  the 
Major  Surface  Combatant  Ships  Project  Manager,    PM  18,    responsibility 
for  T&E  effort  within  the  Naval  Material  Command  relating  to  major 
surface  combatant  ships.      The  ship  classes  include  DLGN  36,    DLGN  38, 
DG,    DD  963,    LHA,    PF,   SCS,   and  CVAN  68. 

The  Notice  recognized  that  in  the  past,    ship  acquisition  T&E 
philosophy  had  been  tailored  to  the  needs  and  production  approach  of 
each  individual  ship  program.     In  light  of  recent  DOD  policy  for  major 
weapon  acquisitions  establishing  more  standard  criteria  for  the  approval 
of  programs  (DCP/DSARC),    a  need  for  adoption  of  more  uniform  T&E 
procedures  was  recognized.      Guidelines  for  test  planning  are  provided 
in  conjunction  with  the  PM-18  functional  organization.     Test  processes 
and  relationships  in  test  design,    development  and  execution  are 
provided. 

The  requirement  for  a  Test  and  Evaluation  Master  Plan  (TEMP) 
as  initial  policy  in  T&E  for  major  surface-combatant  acquisition  pro- 
grams was  laid  on.      The  TEMP  is  to  be  keyed  to  acquisition  milestones 
and  should  cover  all  T&E  requirements  through  the  production  phase 
including  follow-on  operational  testing.      It  should  include  development 
of  the  maintenance  and  ship-overhaul  baseline  test  program  for  the 
equipment  and  system-level  Planned  Maintenance  System  (PMS). 
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It  directed  that  test  planning  should  be  total-ship  oriented  to  assure 
adequate  coverage  of  all  work  breakdown  structure  groups  and  to 
assure  that  all  systems  are  properly  installed  and  completely  operable, 
before  a  ship  is  presented  for  Board  of  Inspection  and  Survey  (BIS) 
trials.     Additionally,    a  Total  Ship  Test  Director  (TSTD)  responsible 
for  planning  and  execution  of  the  T&E  program  is  required  for  each 
program. 

8.  NAVMATINST  3960.6 

NAVMAT  INSTRUCTION  3960.6  of  24  August  1973,    "Planning 
and  Implementation  of  Tests  and  Evaluations  of  New  Weapon  Systems,  " 
provides  further  guidance  and  direction  for  preparing  the  TEMP. 
Under  this  instruction,    the  TEMP  is  considered  to  be  a  management 
document  and  serves  as  a  foundation  for  the  more  detailed  DT&E  and 
IOT&E  plans.     Its  basic  elements  are  objectives,    issues,    responsibil- 
ities,   interfaces  and  schedules.      It  should  be  viewed  as  a  planning 
document  rather  than  a  control  document. 

9.  NAVMATINST  3960.  XX 

NAVMATINST  3960.  XX,    a  draft  instruction  of  21  December 
1973,    is  presently  under  review  within  NAVMAT.      The  instruction, 
"Test  and  Evaluation  of  Ship  Acquisition,  "  has  been  prepared  to 
formalize  NAVMAT  policies  and  procedures  for  ship  acquisition  T&E 
in  order  not  only  to  respond  to  DOD  and  OPNAV  requirements  but 
also  to  provide  coordination  and  standardized  approach  to  the  develop- 
ment of  test  requirements  and  the  supporting  test  documentation. 
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The  Ship  Acquisition  Project  Manager  (SHAPM)  is  assigned 
the  responsibility  for  developing  from  appropriate  OPNAV  design 
requirements  and  his  own  risk  analyses  the  test  requirements  neces- 
sary to  reduce  ship  acquisition  risks. 

Enclosure  (1)  to  this  instruction,    "Management  Plan  for  Total 
Ship  Test  Program  for  Ship  Acquisition,  "  prescribes  the  basic  manage- 
ment structure  and  procedures  to  be  followed  for  developing  test 
documentation  that  will  satisfy  the  objectives  of  ship  acquisition  T&E 
and  will  also  support  the  total  ship  test  program  (TSTP)  for  active 
fleet  ships.      This  enclosure  is  a  comprehensive  document  that  attempts 
to  bring  together  within  a  single  cover  T&E  organization  and  responsibil- 
ities,   test  documentation  requirements,    and  development  of  a  ship 
construction  integrated  test  package.      It  is  against  this  background 
that  the  proposed  and  on-going  T&E  efforts  of  the  Patrol  Frigate 
project  will  be  compared  and  analyzed  in  this  thesis. 
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IV.     SOME  PROTOTYPING  CONSIDERATIONS 

A.      GENERAL 

Navy  prototyping  is  still  evolving  in  the  fly-before-buy  era.     It  is 
based  primarily  on  past  experience  with  prototyping  and  Total  Package 
Procurement.     It  is  also  influenced  by  the  procedural  and  standardiza- 
tion guidance  received  on  a  continuing  basis  from  the  DOD  as  developed 
in  the  previous  chapter.      The  literature  concerning  weapon  system 
acquisition,    DOD  and  Navy  Instruction,    and  various  memoranda  are 
filled  with  positive  statements  concerning  the  use  and  advantages  of 
prototypes.      These  statements  take  the  form  of: 

David  Packard,    "We  want  to  put  more  reliance  on  hardware  and  less 
on  paper  studies  in  advanced  development.  " 

Dr.    John  S.    Foster,    Jr.  ,    "We  desire  to  emphasize  the  use  of 
prototypes.  " 

Barry  J.   Shillito,    "Under  prototype  development,    we  will  evaluate 
hardware  rather  than  paper  promises  in  choosing  a  contractor.  " 
CAPT  E.    J.    Otth,    SHAPM  for  the  Patrol  Frigate  Acquisition,    "We 
expect  to  use  a  land-based  test  site  for  the  combat  system  as  well  as 
the  propulsion  system  to  enable  us  to  verify  integration  and  perform- 
ance of  these  systems  in  an  environment  more  controlled  and  less 
costly  than  aboard  ship.  " 
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Early  in  1974,   discussions  with  Mr.    Matthew  T.    Reynolds, 
NAVMAT's  point  of  contact  in  PM  18  for  all  T&E  matters  relating  to 
major  surface  ships,    indicated  that  trends  in  prototyping  of  combat 
systems  in  new  ship- class  acquisitions  had  reached  the  point  where 
prototyping  was  essentially  mandatory.     At  least  at  this  point  of  time 
in  acquisition  philosophy,    prototyping  is  an  "in"  concept.      Therefore, 
what  prototyping  is  and  what  expected  gains  and  costs  are  incurred  in 
its  use  are  of  considerable  interest. 

Conventional  wisdom  has  it  that  a  prototype  is  a  redundant  and 
often  costly  adjunct  to  the  design  and  development  process.      This  thesis 
considers  the  proposition  that  under  given  circumstances  the  most 
effective  way  of  resolving  some  of  the  uncertainties  of  combat  system 
development  is  by  building  and  testing  a  prototype  system  at  a  land- 
based  test  site.     Underlying  the  discussion  is  the  premise  that  no  single 
specified  pattern  of  behavior  can  adequately  provide  for  all  of  the  con- 
tingencies that  will  arise  in  the  course  of  combat  system  development. 
Additionally,    the  premise  that  cost  and  the  long  lead  and  production 
times  involved  in  ship  construction  do  not  permit  the  prototyping  of 
entire  ships  prior  to  production  decision  points  is  accepted  without 
argument  or  further  comment.      The  combat  system  portion  of  the 
total  ship  is  broken  out  as  a  prime  candidate  for  prototyping  in  contrast 
to  the  relatively  fixed  containment  system  which  houses  and  supports 
the  combat  system. 
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B.      CONCEPTS 

To  many  people,    the  word  "prototype"  evokes  images  of  the  1930's 
and  the   1940's,    suggesting  the  angularity  of  a  biplane,    the  quaintness 
of  goggles,    and  the  hum  of  electronic  equipment  patched  together  with 
tape  and  spliced  wires.      But  what  is  being  discussed  here  is  some- 
thing quite  different;  a  vehicle  for  lessening  technological  uncertainty. 
It  might  be  more  appropriate  to  call  that  vehicle  a  definition-phase  test 
combat  system  or  perhaps  a  demonstrator  combat  system,    rather  than 
a  combat-system  land-based  test  site,    if  only  to  deter  prejudgement 
concerning  the  utility  of  using  a  land  system  to  emulate  shipboard 
systems.      But  prudence  suggests  merely  defining  the  term  as  it  is  to 
be  used  here  and  leaving  language  reform  for  another  occasion. 

There  are  many  possible  themes  in  any  consideration  of  the  use 
of  prototypes  in  development.      The  hypothesis  from  which  this  thesis 
reasons  is  that:     (1)  establishing  the  detailed  configuration  of  a  new 
combat  system  requires  many  decisions,    each  difficult,   and  each 
dependent  on  the  resolution  --  or  avoidance  --  of  uncertainty;  (2)  the 
elemental  substance  of  T&E  is  error  and  uncertainty,    and  delay  in  the 
detection  and  correction  of  error  or  oversight  is  a  principal  cause  of 
inefficiency  or  ineffectiveness  of  development;   (3)  the  early  resolution 
of  uncertainties  in  technology,    objectives,    schedules,    and  costs  is 
vital,    there  are  various  ways  of  resolving  such  uncertainties,    and  a 
technique  of  resolution  most  appropriate  to  the  problem  at  hand  should 
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be  chosen;  and  (4)  in  some  circumstances,    which  can  be  defined,    the 
most  desirable  way  of  resolving  technological  uncertainty  in  ship 
acquisition  programs  may  be  to  build  and  test  a  prototype  combat 
system. 

There  are  many  definitions  of  "prototype"  and  little  profit  in  laying 
them  end  to  end  for  scrutiny.     Basically  this  thesis  will  consider  a 
prototype  combat  system  to  be  a  hardware-software  combination 
approximating,    in  full  scale,    the  main  features  of  a  prospective  opera- 
tional combat  system.     Land-based  tests  must  yield  information  that 
will  permit  the  timely  identification  and  resolution  of  technological 
uncertainties  and  ultimately  support  a  sound  decision  about  the  procure- 
ment of  production  systems.     Additionally,    the-prototype  may  be  used 
to  suggest  ways  the  design,    structure  and  development  of  the  combat 
system  can  best  accommodate  the  shipbuilder's  requirements  for 
detailing  the  design  of  cabling  and  ducting,    assess  maintenance  require- 
ments,   and  be  useful  throughout  the  life-cycle  of  the  production  system 
as  a  test  and  configuration  bed.     Owing  to  the  nature  of  a  prototype,    it 
is  not  very  likely  that  all  prospective  subsystems  will  be  available  as 
early  as  selected  hardware  items,    and  some  simulations  may  have  to 
be  used  in  place  of  other  items. 

C.      WHEN  TO  PROTOTYPE 

The  following  figure  (p.  57  )  presents  some  concepts  for  considera- 
tion in  prototyping  decisions.     Seen  thus,    a  prototype  is  a  device  to 
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moderate  some  of  the  uncertainty  that  characterizes  the  decision 
process.     Because  a  prototype  is  a  full  scale,    semi-operable  representa- 
Hon  of  what  is  intended  to  be  in  time  an  operational  combat  system, 
decisions  buttressed  by  experience  with  prototypes  tend  to  be  founded 
on  less  tenuous  technological  grounds  than  those  based  on  abstract 
analysis  and  derived  from  assumptions.     Reliance  on  prototypes  appears 
to  be  advantageous  for  four  general  classes  of  development  decision. 
(1)  When  there  is  uncertainty  about  the  advisability  of  producing  a 
given  combat  system,    testing  a  prototype  will  provide  reliable  informa- 
tion about  the  attributes  of  the  combat  system  and  will  reduce  the 
quantity  and  importance  of  misgivings.      (2)  When  there  is  uncertainty 
about  which  of  two  similar  hardware/ software  combinations  to  produce, 
testing  subsystems  built  by  different  contractors  will  ease  the  decision. 
In  the  same  vein,    if  cost  overrun's  or  technical  inadequacy  of  subsys- 
tems causes  fall  back  to  other  equipment,    a  prototype  lends  a  suitable 
means  for  incorporating  these  emerging  configuration  changes. 
(3)  When  there  is  uncertainty  about  which  combat  system  to  build, 
testing  prototypes,    liberally  supported  with  realistic  simulations,    will 
provide  hard  information  on  which  a  choice  can  be  based.      (4)  When 
there  is  uncertainty  about  the  technological  feasibility  of  a  configura- 
tion,   prototype  testing  can  assist  in  resolving  the  main  issues. 

The  matter  of  whether  a  prototype  program  is  inherently  more  or 
less  costly  than  a  program  that  bypasses  the  prototype  option  can  be 
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treated,    for  the  moment,    as  incidental.     It  is  apparent  that  cost  and 
schedule  estimates  derived  in  part  from  experience  with  prototype 
combat  systems  will  tend  to  be  somewhat  less  uncertain  than  estimates 
based  solely  on  analytical  predictions  of  development  progress.      From 
the  fact  that  specific  uncertainties  of  technology,    configuration,    and 
probable  mission  responsiveness  will  be  moderated  by  the  very  exist- 
ence of  a  prototype,    it  may  be  adduced  that  a  prototype  strategy  can  be 
used  as  a  means  of  preserving  a  production  option  --  putting  off  a  final 
decision  until  requirements  of  technology  are  better  understood  while 
concurrently  working  toward  that  understanding. 

There  is  no  inherent  reason  why  "Total  Package  Procurement,  " 
should  not  include  a  prototype  phase  if  the  "total  package"  decision 
specified  quantities  and  schedules  but  made  configuration  definition  a 
function  of  an  evaluation  process  following  prototype  testing.      Or,    for 
that  matter,    "total  package"  could  be  interpreted  to  mean  all  develop- 
ment and  production  activity  following  a  definition  phase  that  provided 
for  building  and  testing  prototypes.     Within  current  thinking  however, 
a  prototype  provides  a  vehicle  to  "test  before  flying,  "  and  "flying 
before  buying.  " 

A  prototype  should  be  built  in  the  expectation  of  change,    and  the 
expectation  of  change  is  its  only  substantial  justification.      The  objec- 
tive of  building  a  prototype  is  to  discover  what  changes  are  necessary  - 
what  decisions  must  be  made  --  before  a  design  is  committed  to 
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production,    or  to  discover  if  any  number  of  changes  will  end  in  a 
desirable  production  combat  system.     Such  a  characterization  does 
not  invalidate  the  use  of  prototypes  to  aid  in  resolving  mission  uncer- 
tainty or  to  aid  in  equipment/contractor  selection  or  to  contribute  to  a 
decision  on  which  (rather  than  what)  to  produce,    but  it  emphasizes  the 
contribution  of  a  prototype  to  the  diminution  of  uncertainty,    whatever 
the  nature  of  that  uncertainty. 

From  such  assumptions  arise  two  principles  for  the  conduct  of 
programs  based  on  the  use  of  prototypes:     (1)  controlled  investment  -- 
that  is,    obtaining  an  informational  return  at  an  expenditure  commen- 
surate with  the  worth  of  the  information,    and  (2)  defining  and  limiting 
uncertainty,    which  is  to  say,    having  the  principal  features  of  a  combat 
system  reasonably  well  defined  before  undertaking  construction.     Al- 
though planners  may  squirm  at  the  thought,    there  is  no  obvious  reason 
for  specifying  the  detailed  operational  assignment  of  a  combat  system 
while  a  prototype  is  being  built  and  tested;  the  question  of  whether  it 
should  have  one  or  two  digital  computers  cannot  be  put  off  in  the  same 
way.      Flexibility  however,    must  be  built  into  the  system  to  enable 
modification  to  reasonably  expected  equipment  and/or  threats  in  the 
planning  future. 

A  prototype  combat  system  should  not  be  expected  to  resolve 
uncertainties  that  are  peripheral  to  the  main  purpose  of  building  it, 
nor  should  peripheral  uncertainties  be  permitted  to  dominate  a 
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prototype  program.      Technical  problems  that  can  readily  be  identified 
and  solved  by  analysis  and  discrete  off-site  testing  should  not  be  left 
to  a  prototype  test  inquiry.     Nor  should  very  great  technical  uncertain- 
ties be  left  for  resolution  by  means  of  a  prototype  combat  system. 
Basic  technological  feasibility  is  not  an  issue  that  can  be  satisfactorily 
resolved  at  the  prototype  level;  feasibility  demonstrations  cannot  be 
put  off  until  operational  requirements  begin  to  hinge  on  their  success. 
A  prototype  is  not  a  specific  vehicle  for  uncovering  "unks-unks"  nor 
for  resolving  basic  policy  differences  between  contributing  factions. 
When  the  time  or  urgency  of  a  requirement  is  in  doubt  and  it  is 
possible  to  select  between  two  courses,    one  of  study  and  proposal  and 
the  other  of  build  and  test,    the  relative  advantages  of  the  two  should 
be  very  carefully  evaluated.     If  a  prototype  can  be  built  for  not  much 
more  than  the  cost  of  conducting  an  extended  analysis,    a  prototype 
seems  preferable  because  it  can  be  expected  to  produce  more  reliable 
information  than  a  study.     If  such  information  becomes  available  in 
advance  of  a  production  decision,    the  lead  time  between  production 
decision  and  availability  of  operational  combat  systems  will  be  lessened 
by  the  amount  of  "learning  time"  that  would  otherwise  follow  a  decision 
to  produce  a  combat  system  only  defined  on  paper.     Indeed,    the  per- 
formance of  the  prototype  in  land-based  trials  may  not  only  enhance 
the  validity  of  acquisition  decisions,    it  may  also  solidify  them,    as  a 
decision  based  on  an  existing  prototype  will  generally  contain  less 
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residual  uncertainty  than  one  based  solely  on  analysis  of  engineering 
expectations. 

In  essence,    then,    the  construction  of  prototype  land-based  combat 
systems  can  provide  a  hedge  against  requirements  (strategic)  uncer- 
tainty.    It  does  so  by  permitting  a  more  certain  resolution  of  techno- 
logical uncertainties  than  can  design  studies,    no  matter  how  elegant 
they  are.      The  central  fact  is  that  under  almost  any  conceivable  circum- 
stances the  transition  from  design  proposal  to  first  lot  of  production 
articles  brings  on  change,    and  change  introduces  new  uncertainties. 
Some  are  uncertainties  that  can  be  resolved  with  no  particular  difficulty 
while  production  continues.      The  function  of  a  prototype  is  to  permit 
the  early  identification  of  previously  unrecognized  problems  and  the 
resolution  of  recognized  uncertainties  that  might,    if  they  went  undetec- 
ted,   precipitate  major  changes  in  the  performance,    cost,    or  avail- 
ability of  specific  weapon  systems. 

Adhering  to  the  principles  described  above  would  be  difficult  in 
any  circumstance.     It  is  very  difficult  today.      The  pressures  for  early 
commitment  to  production  are  enormous.      The  customer,    in  this  in- 
stance,   the  Navy,    is  conditioned  by  the  military  environment  to  be 
less  interested  in  the  comprehensive  resolution  of  uncertainty  than 
in  the  early  delivery  of  operational  ships  and  combat  systems.      There 
is  a  natural  tendency  to  assume  that  all  problems  will  be  little  ones. 
In  this  situation  the  producer  has  very  little  motivation  for  investing 
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time  and  money  in  a  prototype.      Profits  come  from  production.      Firms 
are  not  paid  very  well  for  military  development.      Contractors  would 
rather  build  what  are  incorrectly  called  "production  prototypes"  than 
"development  prototypes,  "  because  a  "production  prototype"  implies 
a  strong  commitment  to  production  while  a  "development  prototype" 
represents  a  proposal  of  which  the  faults,    if  any,    are  all  too  evident. 
Here  again,    when  a  "production  prototype"  is  built  the  producer  will 
be  optimistic  about  the  possibility  of  solving  all  problems  early  and, 
as  has  been  observed,    the  customer  is  anxious  to  believe  such  re- 
assurances. 

Some  critics  and  defense-industry  experts   recently  (January  1974) 
have  stated  that  a  prototype,    as  a  preproduction  step,    tends  to  blur  the 
line  between  development  and  full  production,    enabling  the  Services 
to  move  systems  into  production  a  little  bit  at  a  time.     In  another  vein, 
others  argue  that  a  preproduction  prototype  gives  congressional  critics 
more  time  to  attack.      The  presumably  higher  price  tag  is  an  argument 
against  building  the  ship/combat  system  which  in  turn  is  dysfunctional 
to  the  service  who  is  trying  to  abide  by  directives  and  test  before  buy 
concepts. 

D.      DOD,    THE  NAVY  AND  PROTOTYPING 

The  DOD's  current  philosophy  is  that  the  use  of  prototypes  must 
be  increased  in  all  development  efforts  to  reduce  costs  and  resolve 
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uncertainties  prior  to  major  production  commitments.      DOD  has 
broken  prototypes  into  three  categories  as  follows: 

1.  Experimental  --to  demonstrate  technological  feasibility. 

2.  Developmental  --to  gain  sufficient  confidence  before 
comitting  a  system  to  full-scale  development. 

3.  Production  --to  prove  out  the  system,    tools  and  methods 
prior  to  quantity  production. 

The  DOD  normally  recognizes  three  approaches  to  prototyping, 
which  are  in  most  cases  applied  to  experimental  and  developmental 
prototypes.     They  are  as  follows: 

1.  Sole/single  source  --  prototypes  developed  and  fabricated 
by  a  single  developer. 

2.  Competitive  --  prototypes  involving  the  same  technology, 
developed  and  fabricated  by  two  or  more  developers.  ti 

3.  Complementary  --  prototypes  involving  different  tech- 
nologies,   developed  and  fabricated  by  two  or  more  developers  in 
response  to  the  same  basic  requirement. 

The  breakdown  and  extent  of  prototyping  per  se  is  broken  out  as 
follows : 

1.  Component  --  part  of  an  assembly  of  parts  to  perform  a 
function. 

2.  Subsystem  --a  major  functional  assembly  of  a  system. 
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3.  Hybrid  --a  system  of  equipment  based  upon  an  alteration 
or  change  to  an  existing  system  or  equipment. 

4.  "Bare-bones"  model  --  a  system  or  equipment  containing 
only  its  minimum  essential  elements. 

5.  Complete  system  --a  combination  of  subsystems  that 
makes  up  the  ultimate  system  or  equipment. 

As  should  be  seen  from  the  above,    prototyping  can  take  many  forms 
and  seek  many  ends.      To  achieve  a  completely  successful  and  highly 
efficient  set  of  prototype  terminology  is  one  of  the  important  goals 
which  the  Navy  feels  must  be  achieved,    if  it  is  to  be  successful  in  im- 
plementing recent  prototyping  policies.     The  figure  below  depicts  what 
appears  to  be  the  direction  the  Navy  is  going  in  its  efforts  to  comply 
with  guidance  from  DOD  on  prototyping  and  defense  system  acquisition 
in  achieving  a  set  (or  body)  or  prototype  terminology  which  can  be 
utilized  through  the  spectrum  of  Navy  acquisition. 


TERMINOLOGIES 


Prototype  Weapon 

Terminology         Subsystems 


Ships-conventional      Ships-new 
hull  design  hull  design 


Experimental        Bread  board 
Prototypes  Models 


not 
applicable 


Research 
Vehicles 


Development         Advanced 
Prototypes  development 

models/ 
prototypes 


-land -based 

testing 
-test  bed  ships 


Test  craft 


Production 
Prototypes 


Pilot  production        Lead  ship 
models/proto- 
types  


Lead  ship 


65 


i       \ 


A  word  of  caution  is  in  order  when  applying  any  so-called  guide 
such  as  that  in  the  previous  figure.     The  purpose  of  such  a  guide  is 
to  enhance  understanding  and  communication,    not  to  impose  a  rigid  set 
of  rules  which  will  do  more  harm  than  good.      The  best  example  avail- 
able is  probably  that  of  shipbuilding,    whereby  a  period  of  several  years 
will  be  required  for  transition  from  present  methods  of  shipbuilding  to 
the  sequential  "fly-before-buy"  approach,    even  for  ships  in  the  5,000-ton 
category.      For  larger  ships  such  as  carriers,    the  Navy  must  and  will 
look  for  innovative  ways  to  comply  with  the  spirit  of  new  prototype,    test, 
evaluation  and  acquisition  policies,    but  an  idealized  sequential  program 
plan  will  not  be  attainable  in  all  cases. 

Likewise,    the  figure  should  not  be  interpreted  to  mean  that  each 
and  every  step  must  be  accomplished  in  each  and  every  program.     As 
a  general  rule,    when  determining  whether  certain  steps  may  be  omitted 
or  not,    development  prototypes  should  not  be  required  when  the  tech- 
nology has  been  "hardware"  demonstrated  in  some  other  program  or  by 
experimental  prototypes.     Neither  should  production  prototypes  be 
required  when  engineering  change  proposals  of  a  minor  nature  can  be 
made  to  "off  the  shelf"  or   "in  production"  defense  systems  of  another 
Service. 

F.      SUMMARY 

The  central  thesis  of  this  chapter  is  that  in  certain  circumstances 
it  is  sensible  to  build  and  test  a  prototype  of  a  combat  system  before 
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finally  deciding  to  produce  it  in  quantity.     Such  circumstances  can  occur 
if  the  technological  risks  inherent  in  the  design  are  substantial  and  can- 
not be  reduced  to  manageable  proportions  by  analysis  alone.      The  dif- 
ficulty of  deciding  when  to  adopt  a  prototype  strategy  arises  from  the 
necessity  of  deciding  a  priori  how  substantial  are  the  technological, 
mission  assignment,    or  source-selection  risks. 

The  circumstances  under  which  the  construction  of  a  prototype 
combat  system  becomes  a  desirable  element  of  the  total  development 
program  include: 

1.  A  program  sufficiently  well  defined  to  permit  developers 
to  undertake  the  resolution  of  specific  technological  uncertainties  that 
cannot  realistically  be  expected  to  succumb  to  alternative  techniques 
of  uncertainty  resolution  (such  as  analysis  and  simulations)  or  that 
can  be  alternatively  resolved  only  at  greater  cost  than  by  the  prototype 
route.      (Build  a  complete  ship  only  to  discover  that  the  combat  system 
doesn't  operate  properly.  ) 

2.  When  developers  understand  that  the  real  purpose  of  a 
prototype  combat  system  is  to  identify  necessary  design  changes 
quickly  and  accurately,    and  that  one  outcome  of  a  prototype  program 
can  be  a  decision  not  to  proceed  to  production.      The  object  of  a  proto- 
type is  to  permit  a  prompt  and  economical  decision  in  such  matters. 

The  real  payoff  which  the  Navy  expects  to  achieve  in  prototyping 
programs  can  only  be  realized  if  thorough  cost  analysis  and  test  and 
evaluation  is  conducted,    reported  and  utilized  by  source  selection 
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boards,    project  managers,    contract  negotiators,    and  high-level 
decision  makers  at  every  sequential  step  in  the  defense  system  ac- 
quisition process.     The  remainder  of  this  thesis  explores  the  efforts 
of  the  Patrol  Frigate  program  to  accomplish  these  objectives. 
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V.     THE  PATROL  FRIGATE'S  TEST  AND  EVALUATION  PROGRAM 

A.  GENERAL 

The  Patrol  Frigate  Ship  Acquisition  Program  was  chosen  as  an 
example  of  ongoing  T&E  and  prototyping  efforts  because  of  the  avail- 
ability of  information,    the  cooperation  extended  by  Project  Office 
personnel,   and  several  unique  features  of  the  acquisition  program 
itself  which  lend  application  to  subsequent  ship  procurement  programs. 
The  PF  is  also  the  first  major  ship  acquisition  program  to  come  under 
the  auspices  of  DODI  5000.  1,    DODI  5000.  3,    and  additional  T&E  guide- 
lines developed  in  Chapter  3. 

B.  PATROL  FRIGATE  SHIP  HIGHLIGHTS 
1.       Objective  of  the  PF  Program 

The  objective  of  the  PF  program  is  to  acquire  for  the  late 
1970's- 1980's  timescale  a  class  of  ships  capable  of  ensuring  our  use 
of  essential  sea  lines  of  communications.     It  is  intended  that  these 
ships  will  provide,    at  least  cost,    the  necessary  improvement  in  the 
Navy's  surface  combatant  capability  to: 

Defend  non-carrier  forces  against  airborne  forces 
Conduct  ASW  operations 
These  operations  will  be  conducted  in  conjunction  with  other 
sea  control  forces. 
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2.  Ship  and  Program  Constraints 

The  PF  is  an  austere  escort  ship  designed  toward  the  major 
program  goals  initially  imposed  by  CNO,   as  follows: 

Average  follow  ship  cost  (without  escalation)  not  exceeding 
$45.7  million  (FY  73  dollars). 

Full  load  displacement  not  exceeding  3400  tons 
Total  personnel  accommodations  not  exceeding  185. 

In  order  to  meet  the  cost  constraints,    weapons  and  sensors 
requiring  research  and  development  have  not  been  included  in  the  PF 
design,    at  least  in  theory.      Ensuring  approval  for    service  use  of  major 
individual  equipments  and  systems  has  been  made  the  responsibility  of 
the  Systems  Commanders,    NAVSHIPS,    NAVORD,    etc. 

It  is  intended  that  the  PF  will  operate  in  conjunction  with  other 
sea  control  forces  and  not  as  an  independent  unit.     Hence  the  PF  has     , 
been  designed  to  supplement  the  capability  of  other  ships  rather  than 
operate  in  a  stand-alone  capacity. 

3.  Mission  Requirements 

To  ensure  our  use  of  essential  sea  lines  of  communication 
against  the  contemporary  and  postulated  threats  along  with  planned 
and  existing  escort  forces,    Development  Concept  Paper  #97  of  3  April 
1973  requires  the  PF  to  be  able  to: 

Provide  self-defense  and  operate  as  part  of  an  ASW 
screen  defending  the  escorted  force  or  operate  on  picket  station  in 
support  of  the  escorted  forces. 
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Operate  in  conjunction  with  other  escorts  to  provide  area 
and  close-in  AAW  defense  of  the  escorted  force  and  self-defense 
against  aircraft  and  airborne  anti-shipping  missiles. 

Provide  for  defense  of  the  escorted  force  and  own  ship 
against  hostile  surface  ships  to  over-the-horizon  ranges. 

Operate  in  conjunction  with  other  forces  in  open  ocean 
ASW  operations  and  subsurface  surveillance. 

Supplement  area  AAW  forces  in  the  detection  and  engage- 
ment of  airborne  threats. 

Enforce  blockades  and  operate  offensively  in  conjunction 
with  other  forces  against  mercantile  shipping  and  naval  forces. 

Provide  for  SAR,    surveillance  and  evacuation  operations 
and  establish  a  naval  presence. 

t    -         The  PF  is  not  intended  to  serve  as  a  carrier  force  escort 
or  in  high-threat  areas. 

4.       PF  Combat  System  Components 

Systems   selected  for  the  PF  consist  of  the  following: 
WEAPON 

-Standard  Missile  (Surface-to-air) 
-Harpoon  Missile  (Surface-to-surface) 
-76mm  Oto    Melara  Gun 

-Space  and  weight  for  Close  In  Weapons  System  (CIWS) 
-Mk  46  Torpedo 
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LAUNCHER 

-Mk  13  Guided  Missile  Launcher  (Fires  Harpoon  or  Standard  Missiles) 

-Mk  32  Torpedo  Tubes 

FIRE  CONTROL  SYSTEM 

-Mk  92  (STIR)  Two  Channel  System 

SENSOR 

-SPS-49  Air  Search  Radar 

-SPS-55  Surface  Search  Radar 

-SQS-56  Sonar 

-Passive  ESM  System 

COMBINED  SENSOR  AND  WEAPONS  DELIVERY 

-Two  LAMPS  Helicopters 

The  required  capability  of  the  above  systems  elements  has 
-  -beenpreviously  established  and  it  is  required  that  their  integration  .-- 
into  a  single  combat  system  shall  in  no  way  adversely  alter  these 
established  capabilities. 

5.       PF  Procurement  Phasing 

A  contract  has  been  awarded  to  Bath  Iron  Works  Corporation 
to  build  the  lead  ship.      Contracts  for  follow- ship  procurements  are 
tentatively  to  be  in  two  batches  of  24  and  25   each  over  a  five-year 
period.     Ship  deliveries  are  to  be  at  the  rate  of  about  one  per  month, 
with  the  lead  ship  being  delivered  in  June   1977  and  the  last  ship  of  the 
second  block  in  the   1981-1982  timeframe. 
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C.      PATROL  FRIGATE  T&E  PROGRAM 

The  T&E  phases  for  the  PF  are  considered  to  be  Developmental 
Test  and  Evaluation  (DT&E),    Operational  Test  and  Evaluation  (OT&E) 
and  Production  Acceptance  Test  and  Evaluation  (PAT&E).      OT&E  is 
further  broken  down  into  Initial  Operational  Test  and  Evaluation  (IOT&E) 
and  Follow-on  Test  and  Evaluation  (FOT&E). 

DT&E  is  viewed  as  a  class  of  tests  and  evaluations  for  the  specific 
purpose  of  facilitating  the  evolution  of  the  ship  system  and  is  conducted 
under  the  sponsorship  of  the  Developing  Agency.      For  the  PF,    DT&E 
is  that  portion  of  land-based  test  site  testing  concerned  with  installation 
and  integration  of  the  combat  systems  in  addition  to  that  developmental 
testing  to  be  conducted  in  support  of  the  individual  equipments.     Integra- 
tion of  the  combat  system  is  considered  to  be  a  "Risk  Watch  List"  item, 
i.e.,  -failure  to  achieve  satisfactory  integration  at  the  test  site  would 
adversely  affect  DSARC  III  presentation.     Integration  of  the  Combat 
System  is  to  be  proved  at  the  test  site  prior  to  DSARC  III. 

IOT&tE  for  the  PF  consists  of  two  separate  areas.      First,    the 
individual  equipment/systems  of  the  "Risk  Watch  List"  in  the  DCP 
are  to  undergo  separate  IOT&E's.      For  instance,    the  Mk  92  fire  con- 
trol system  is   scheduled  to  undergo  IOT&E  on  the  USS  TALBOT  DEG-4 
in  September  of  1974.     Second,    the  integrated  systems  will  undergo 
IOT&E  at  the  land-based  test  site.     "Risk  Watch  List"  systems  which 
will  undergo  individual  IOT&E  are: 
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-Mk  92   FCS/MK  75  gun  mount  and  76  mm  ammunition 

-AN/SPS-49  Radar 

-AN/SQS-56  Sonar 

-HARPOON  Fire  Control  System 
The  responsibility  for  initiating  IOT&E  for  these  systems  lies 
with  the  participating  managers   (FARM'S)  managing  the  development 
and  procurement  of  their  respective  equipment.     However,    the  T&E 
Manager  of  the  SHAPM's  staff  will  closely  monitor  the  test  programs 
to  insure  that  PF  test  schedules  can  be  met  and  that  any  problems  or 
deficiencies  during  the  IOT&E's  are  taken  into  account. 

IOTkE  of  the  integrated  systems  at  the  Combat  System  Land- 
Based  Test  Site  (CSLBTS)  is  scheduled  to  begin  in  January  1975. 
Naval  personnel  will  be  used  to  evaluate  the  systems.     Accomplish- 
ment of  adequate  IOT&E  at  the  LBTS  with  satisfactory  results  is  a  . 
prerequisite  for  approval  by  DSARC  of  follow- ship  procurement.     As 
such,    the  events  and  planning  leading  to  IOT&E  as  well  as  IOT&E 
itself  will  receive  the  closest  attention  of  the  OT&E  Organization. 

FOT&E  will  be  addressed  in  detail  by  an  FOT&E  Plan  which  will 
include  the  possible  requirements  and  procedures  for  assigning  the 
lead  ship  to  OPTEVFOR  for  a  complete  at-sea  operational  appraisal 
prior  to  the  ship's   release  for  Fleet  usage.      This   requirement  was 
embodied  in  the  DSARC  I/II  decision  of  27  September   1972  which 
stated  in  part: 
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The  Navy  is  requested  to  develop  a  plan  for,    and  evaluate  the 
impact  of  assigning  the  lead  PF  to  OPTEVFOR  for  a  reasonable  period 
to  complete  an  at-sea  operational  appraisal  of  the  PF  as  a  whole  prior 
to  the  lead  ship's  full  release  for  Fleet  usage.      This  plan  and  evalua- 
tion,   together  with  the  Navy's  recommendations,    should  be  submitted 
to  OSD  at  the  time  of  preparation  of  the  revision  to  the  DCP  for  initia- 
tion of  construction  of  the  first  follow  ship. 

The  impact  of  this  assignment  will  be  evaluated  and  an  evaluation 
report  will  be  prepared  by  the  T&E  Planning  Group  for  submission  to 
the  CNO.      The  FOT&E  Plan  and  the  evaluation  report  will  be  com- 
pleted prior  to  the  DSARC  III  milestone. 

PAT&E  for  the  PF  is  functionally  separated  into  three  areas. 
These  areas  are  documentation  certification  at  the  LBTS's,    acceptance, 
testing  and  documentation  validation  on  the  lead  ship,    and  acceptance 
testing  on  the  follow  ships.     Detailed  information  on  the  PAT&E  is 
contained  in  the  Ship  Test  Management  Plan  of  15  November  1972. 

Documentation  certification  at  the  LBTS's  will  consist  of  actual 
conduct  of  test  procedures  for  the  installed  equipments /systems. 
Such  use  will  enable  test  procedures  to  be  corrected  if  needed  and 
certified  prior  to  use  on  the  lead  ship.      More  will  be  said  about 
"certification"  in  a  later  section  of  this  thesis. 

On  the  lead  ship,    the  certified  procedures  will  be  used  to  conduct 
the  tests  leading  to  validation  of  the  procedures  and  acceptance  of 
the  shipbuilder's  efforts.      Test  procedures  for  those  equipments  and 
systems  which  were  not  certified  at  the  LBTS  will  also  be  validated 
on  the  lead  ship.      The  validated  documentation  from  the  lead  ship 
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will  then  be  supplied  to  follow  shipbuilders.      Follow-ship  testing  will 
use  validated  test  procedures  for  test  conduct  leading  to  Navy  accept- 
ance of  the  shipbuilder  effort. 

i 

D.      PATROL  FRIGATE  TEST  ORGANIZATION 

The  Patrol  Frigate  Ship  Acquisition  Project  Manager  has  established 
a  Total  Ship  Test  Program,    and  the  organization  for  its  accomplishment 
for  the  Patrol  Frigate.      The  organization  that  will  be  used  by  the  Patrol 
Frigate  to  develop  test  documentation  is  shown  in  Figures   1  and  2.     The 
principal  members  of  the  organization  are  identified  in  this   section. 
Specific  responsibilities  are  elaborated  in  the  next  section. 

1.  Ship  Acquisition  Project  Manager 

The  Ship  Acquisition  Project  Manager  (PMS-399)  is  respon- 
sible to  the  Commander,    Naval  Ship  Systems  Command  and  the  Major 
Surface  Combatant  Ship  Project  Manager  (PM-18)  for  the  Patrol  Frigate 
Ship  Acquisition. 

2.  Test  and  Evaluation  Manager 

The  Test  and  Evaluation  Manager  (PMS-399T)  is  responsible 
to  the  SHAPM  for  the  successful  conduct  of  the  PF  Total  Ship  Test 
Program. 

3.  Total  Ship  Test  Director 

The  Supervisor  of  Shipbuilding  at  Bath,    Maine  has  been 
designated  as  the  Total  Ship  Test  Director  and  is  responsible  for  the 
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development  of  ship  test  documentation  and  its  validation  on  the  Lead 
Ship. 

4.       Engineering  System  Test  Development  Director 

The  Naval  Ship  Engineering  Center,    Philadelphia  Division  has 
been  designated  as  the  Engineering  System  Test  Development  Director, 
with  the  responsibility  for  the  development  of  the  Hull,    Machinery  and 
Electrical  (H/M/E)  test  documentation. 

.  5.       Combat  System  Test  Development  Director 

Naval  Ships  Weapon  System  Engineering  Station,    Norfolk 
Detachment  has  been  designated  as  the  Combat  System  Test  Develop- 
ment Director  with  overall  responsibility  for  the  development  of  Com- 
bat System  test  documentation. 

6.  Test  Development  Managers 

The  Test  Development  Managers  under  direction  of  the  Combat 
System  Test  Development  Director,    provide  test  documentation  for 
specific  subsystems  of  the  Combat  System. 

7.  Combat  System  Integration  Manager 

The   Combat  System  Integration  Manager,    PMS-399.  13,    is 
responsible  for  successful  integration  of  the  elements  of  the  PFCS 
and  the  provision  of  system  level  test  documentation  for  the  testing 
of  the  lead  and  follow  ships. 

8.  Officer-In- Charge  Combat  System  Land- Based  Test  Site 
The  Officer-In- Charge  of  the  Combat  System  Land  Based 

Test  Site  coordinates  activities  at  the  Combat  System  Land  Based 
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Test  Site  with  specific  responsibilities  as  described  in  the  next 
section. 

9.       Lead  Shipbuilder 

Bath  Iron  Works   (BIW)  has  been  selected  as  the  Lead  Ship- 
builder.     In  that  capacity  the  preparation  of  various  items  of  test 
documentation  and  ship  test  performance  are  within  their  responsibility. 
10.       Combat  System  Integration  Support  Contractor 

The  Naval  Ship  System  Command  has  entered  into  a  contract 
with  Sperry  Systems   Management,    Sperry  Rand  Corporation,    as  the 
Combat  System  Integration  Support  Contractor  to  assist  the  Combat  , 
System  Integration  Manager  (CSIM)  in  the  providing  of  system  level 
test  documentation. 

E.      PATROL  FRIGATE  TEST  ORGANIZATION 
1.       Introduction 

The  PF  SHAPM  has  established  a  Total  Ship  Test  Program 
for  the  Patrol  Frigate  in  accordance  with  existing  regulations  and 
which  appears  in  the  Patrol  Frigate  Ship  Test  Management  Plan.     This 
program  is  to  provide  for  comprehensive  tests  which  will  determine 
the  status  of  equipment  readiness  during  construction  and  the  life-cycle 
of  the  ship.      The  organization  that  will  be  used  by  the  PF  to  develop 
test  documentation  is  identified  in  the  previous  section.     Detailed 
responsibilities  of  the  actors  involved  in  T&E  of  the  PF  as  presently 
viewed  follows. 
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2.  Ship  Acquisition  Project  Manager 

The  SHAPM  provides  guidance  as  necessary  to  the  T&E 
Manager  in  the  areas  of  planning,    budgeting,    developing,    and  apprais- 
ing ship  test  documentation.     The  SHAPM  is  responsible  for  the  PF 
Ship  Acquisition  Program  and  will  provide  guidance  as  necessary  to 
the  T&E  Manager  for  the  implementation  of  the  Total  Ship  Test  Program 
(TSTP). 

3.  Test  and  Evaluation  Manager 

The  Test  and  Evaluation  Manager  (PMS-399T)  is  responsible 
for  the  successful  conduct  of  the  Patrol  Frigate  TSTP  with  specific 
responsibilities  for: 

-Successful  conduct  of  the  T&E  Program 

-Generation  of  the  test  sections  of  the  Follow  Ship 
Specifications 

-Test  of  the  Lead  and  Follow  Ships 

-Development  and  maintenance  of  the  Test  and  Evaluation 
Master  Plan  and  Ship  Test  Management  Plan 

-Analysis  and  feedback  of  the  results  of  testing  into  ship 
design 

-Development  and  maintenance  of  documentation  for  the 
testing  of  Lead  and  Follow  Ships 

-Compliance  with  T&E  policy  established  by  the  Major  Sur- 
face Combatant  Ship  Project  Manager  (PM-18)  and  the 
Commander,    Naval  Ship  Systems   Command. 
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4.       Total  Ship  Test  Director  (TSTD) 

The  Total  Ship  Test  Director  is  responsible  to  the  SHAPM 
through  the  T&E  Manager  for  coordinating  the  development  ship  test 
documentation  and  its  validation  on  the  Lead  Ship. 

The  TSTD  has  specific  responsibilities  to: 

-Manage  the  development  of  an  Integrated  Test  Package  to 
ensure  that  the  Follow  Shipbuilders  comply  with  the  Ship- 
building and  Performance  Specifications. 

-Manage  the  development  of  a  test  package  for  the  life-cycle 
testing  of  the  PF. 

-Prepare  and  forward  Quarterly  Progress  Reports  and 
Exception  Reports. 

-Supervise  the  conduct  of  testing  of  the  Lead  Ship  during 
construction  at  Bath  Iron  Works. 

-Validate  Test  Procedures  during  the  testing  of  the  Lead  Ship. 

-Direct  the  review  and  revision  of  the  test  sections  of  the 
Lead  Shipbuilding  Specifications  and  the  Test  Documentation 
Booklet  for  use  in  the  Follow  Ship  Request  for  Proposal. 

-Direct  the  activities  of  the  Combat  System  and  Engineering 
System  Test  Development  Directors  in  accomplishing  the 
above  tasks. 

-Submit  schedule  and  budget  data  to  the  PF  T&E  Manager. 

-Maintain  a  configuration  control  program  to  ensure  avail- 
ability of  correct  test  procedures  for  Lead  and  Follow  Ship 

testing. 
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5.  Engineering  System  Test  Development  Director  (ESTDD) 
The  ESTDD  has  specific  responsibilities  to: 
-Accomplish  tasks  assigned  by  the  TSTD  for  development  of 

the  Test  Index  and  Test  Outlines  for  Contractor   Furnished 

Equipment  for  the  Lead  and  Follow  Ship  Test  Programs. 
-Provide  technical  assistance  for  liaison  and  monitoring 

during  Engineering  System  test  documentation  validation  on 

the  Lead  Ship. 
-Many  other  duties  relating  to  H/M/E  tests  beyond  the  scope 

of  this  thesis. 

6.  Combat  System  Test  Development  Director  (CSTDD) 
The  CSTDD  has   specific  responsibilities  to: 
-Accomplish  tasks  assigned  by  the  TSTD  for  development  of 

Combat  System  test  documentation  for  the  Lead  and  Follow 

Ship  Test  Programs. 
-Provide  for  on-site  liaison  and  monitoring  during  Combat 

System  test  documentation  validation  on  the  Lead  Ship. 
-Review  the  Integrated  Test  Package  and  recommend  which 

Combat  System  Test  Procedures  are  suitable  for  conversion 

into  Maintenance  Requirement  Cards  for  inclusion  in  the 

Planned  Maintenance  Subsystem. 
-Provide  for  on-site  witness  of  the  conduct  of  tests  at  the 

CSLBTS  and  certifying  the  correctness  of  Test  Procedures. 
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-As  directed  by  the  TSTD,    participation  in  the  review,    and 
recommend  revision,    of  the  test  sections  of  the  Lead  Ship- 
building Specifications  and  the  Test  Documentation  Booklet 
applicable  to  the  Combat  System  for  use  in  the  Follow  Ship 
Request  for  Proposal. 

-Monitor  development  of  Combat  Systems  Test  Procedures  and 
outlines. 

-Review  all  Combat  System  test  documentation  to  ensure  its 
technical  accuracy  and  adequacy  to  support  the  ship  test 
process. 

-Prepare  schedule  and  budget  recommendations  for  the  Lead 
Ship  Test  Program. 
7.       Test  Development  Managers  (TDMs) 


The  TDMs  are  responsible  for  organizing,    planning,    managing, 
and  controlling  the  development  of  specific  subsystem  test  documentation 
for  the  Combat  System.      The  TDMs  are  responsible  for: 

-Managing  the  development  of  all  equipment  and  subsystem 

test  procedures  for  equipment  under  their  cognizance. 
-Reviewing  Test  Procedures  for  technical  accuracy  and 

content. 
-Managing  the  development  of  special  and  at-sea  trial  Test 

Procedures  for  external  communications,    ASW  systems, 

LAMPS,    and  Harpoon  Weapon  System,    where  applicable. 
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-Ensuring  the  generation,    review,    and  modification  of  the 
Test  Index,    Test  Outlines,    and  Test  Sequence  Networks  for 
equipment  under  their  cognizance. 

-When  directed  by  the  CSTDD,    acting  as  certifying  agents  at 
the  CSLBTS  and  validating  agents  on  the  Lead  Ship  for  the 
certification  and  validation  of  test  documentation  for  equip- 
ments under  their  cognizance. 

-Providing  solutions  to  test  problems  encountered  during  the 
conduct  of  testing  at  the  CSLBTS  and  onboard  the  Lead  Ship. 
The  Officer-In- Charge  (OIC)  CSLBTS  and  the  TSTD  have 
been  given  authority  by  the  SHAPM  to  make  interim  changes 
to  Test  Procedures  during  testing  at  the  CSLBTS  and  onboard 
the  Lead  Ship  respectively. 
8.       Combat  System  Integration  Manager  (CSIM) 

The  CSIM's  principal  function  is  to  insure  the  successful 
integration  of  the  PF  Combat  System.     His  primary  responsibilities  in 
support  of  the  Ship  Test  Program,    are  as  follows: 

-Provision  of  special  and  at-sea  trial  Test  Procedures  for 
certification  at  the  CSLBTS  and  for  eventual  Lead  and  Follow 
Ship  testing  as  applicable. 

-Revision   of  CSLBTS  Test  Procedures  for  Lead  Ship  applicabil- 
ity and  generation  of  additional  Test  Procedures  for  Lead 
Ship  as   required. 


*  *— 
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-Development  of  Test  Procedures  for  Combat  System  com- 
ponents installed  in  Follow  Ships  and  not  in  the  Lead  Ship. 
9.       Lead  Shipbuilder 

The  Lead  Shipbuilder  is  responsible  for  the  following  Ship 
Test  actions  in  accordance  with  his  Contract: 

-Review  the  Test  Documentation  Booklet  and  prepare  revisions 
to  equipment,    subsystem,    system  and  special  test  outlines  as 
necessary  to  ensure  accuracy  and  adequacy  of  Ship  Test 
Program. 

-Develop  Test  Procedures  for  Contractor  Furnished  Standard- 
ized equipments. 

-Develop  Test  Memoranda  for  CFE. 

-Conduct  Combat  System  equipment,    subsystem,    system  and 

special  test  procedures  onboard  the  Lead  Ship.     Deficiencies 

shall  be  brought  to  the  attention  of  the  SUPSHIP,    Bath. 

10.       Officer-In- Charge  Combat  System  Land-Based  Test  Site 
(QIC  CSLBTS) 

The  OIC  CSLBTS  is  responsible  to  the  SHAPM,    through  the 

Technical  Director,    for  the  overall  management  and  operation  of  the 

Navy  assets  at  the  CSLBTS.     In  support  of  the  Ship  Test  Program,    he 

has  the  responsibility  to: 

-Demonstrate,    where  feasible,    Combat  System  test  procedures 

at  the  CSLBTS  for  certification  prior  to  use  on  the  Lead  and 

Follow  ships. 
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-Schedule  tests  and  demonstrations. 

-Supervise  the  evaluation  of  proposed  changes  to  Combat 
System  Test  Procedures  at  the  LBTS. 
11.       Combat  System  Integration  Support  Contractor 

Test  responsibilities  of  Sperry  Systems  Management  as  the 
Combat  System  Integration  Support  Contractor  includes: 

-Develop  unique  subsystem,    system  and  special  Test  Pro- 
cedures for  the  Combat  System,    initially  for  use  at  the 
LBTS  and  eventually  for  use  during  the  Ship  Test  Program. 

-Provide  maintenance  assistance  and  engineering  support 
during  certification  of  equipment,    subsystem,    system  and 
special  Combat  System  Ship  Test  Procedures  at  the  LBTS. 

-Evaluate  proposed  changes  to  Combat  System  Test  Pro- 
cedures at  the  LBTS  and  forward  results  with  recommenda- 
tions to  the  TSTD  via  the  CSTDD. 

-Provide  the  TSTD  technical  support  if  required  during  the 
conduct  of  Test  Procedures  on  the  Combat  System  for  both 
the  Lead  Ship  and  Follow  Ships. 
Highlights  of  the  present  Navy  contract  with  Sperry  Systems 
Management  will  be  provided  in  a  subsequent  chapter  of  this  thesis. 

F.      CONCLUSIONS 

Requirements  set  forth  in  DODD  5000.  1  and  current  Navy  Depart- 
ment Directives  provide  the  required  Patrol  Frigate  test  and  evaluation 
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guidance  with  which  to  establish  a  ship/system  T&E  approach  which 
demonstrates,    at  various  decision  points,    sufficient  confidence  for 
follow- ship  commitments.      The  PMS  399  organization  for  T&E  has  the 
capability  of  providing  for  successful  acquisition  of  50  mission- capable 
Patrol  Frigates  with  performance-validated  ship's  crews,    ship's 
hardware  and  software. 

Effective  implementation  of  the  PF  T&E  program  concept  requires 
centralized  authority  and  responsibility  in  one  agency  acting  for  and 
under  the  direction  of  the   SHAPM.      This  has  been  provided  by  a 
Test  and  Evaluation  Manager.      One  caveat  however,    is  in  order  and 
is  in  reference  to  behavioral  considerations  of  participative  manage- 
ment as  employed  by  the  PF  T&E  organization.      The  SHAPM's   T&E 
organization  is  predominately  a  military/ civil  service  organization 
and  its  managers  have  had  a  lifetime  in  military  and  government  affairs., ,.« 
In  such  organizations  it  is  often  a  rule  not  to  bring  up  problems  that  do 
not  already  have  apparent  solutions.      The  essence  of  participative 
management  in  the  PF  T&E  program  is  openness  and  freedom  of 
exchange  of  both  problems  and  problem  solutions.      Centralized  top 
management  seeks  this,    but  it  does  not  come  naturally  and  requires 
perceptive  treatment  by  all  who  are  involved. 

Another  area  in  which  the  PF  program  is  vulnerable  stems  from 
the  ever  increasing  number  of  Congressional,    DOD  and  Navy -mandated 
programs.      Previous  chapters  have  dealt  with  T&E  aspects.     Similar 
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forces  are  at  work  in  other  areas  such  as  integrated  logistic  support, 
value  engineering,    surplus  labor  area  selection,    small  business  with- 
holds and  shipbuilding.     Across  the  board  PMS  399  is  particularly- 
vulnerable  to  those  programs  which  are  aimed  at  increasing  centraliza- 
tion of  weapon  system  procurement.     While  PMS  399  will  support  these 
programs,    they  have  a  significant  impact  on  planning  because  they  are 
often  of  great  scope,   unfunded,    absorb  scarce  and  crucial  resources, 
and  decrease  the  SHAPM's  planning  self-determination. 
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VI.     THE  PATROL  FRIGATE  COMBAT  SYSTEM 
LAND-BASED  TEST  SITE 


A.      GENERAL 

Land-based  integration  of  the  Patrol  Frigate  combat  system  is  a 
keystone  of  the  PF  T&E  program.     Prior  to  the  production  decision 
(DSARC  III),    presently  scheduled  for  March  of  1975,    the  Land-Based 
Test  Site  will  be  used  to  prove  installation  procedures  to  the  maximum 
extent  practicable  and  to  prove  the  mechanical  and  electrical  compat- 
ibility of  the  major  equipments  of  the  combat  system.     Additionally 
the  Combat  System  Preliminary  System  Operational  Program  will  be 
demonstrated  at  the  CSLBTS.      The  Final  Operational  Program  will  be 
debugged  at  the  LBTS,    after  DSARC  III,    and  will  be  demonstrated  on- 
board the  lead  ship  after  delivery. 

Proofing  of  computer  programs  will  be  emphasized  as  a  major 
factor  in  achieving  full  utilization  of  the  integrated  combat  system. 
Engineering  and  functional  analysis  at  the  LBTS  will  lead  to  early 
solution  of  integration  problems  that  may  be  exposed. 

An  attempt  will  be  made  to  address  critical  test  and  evaluation 
questions  and  issues  identified  in  the  DCP  such  as: 

-Are  the  individual  combat  subsystems  which  are  to  be  incor- 
porated in  the  ship  technically  acceptable  and  operationally  suitable 
and  effective? 
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-Is  the  integration  of  the  various  combat  systems  through  their 
respective  interfaces  adequate,    in  terms  of  data  transfer  characteristics 
such  as  handling  capacity,    rate  and  quality,    to  meet  operational 
requirements? 

-Do  those  operational  characteristics  of  the  integrated  combat 
system  which  can  be  estimated  based  on  initial  operational  test  and 
evaluation,    including  reliability  and  maintainability  test  and  evaluation, 
show  a  reasonable  probability  that  the  ship  class,    when  afloat,    will  be 
able  to  satisfactorily  accomplish  the  mission  for  which  it  was  designed? 

The  use  of  a  land-based  test  site  as  an  aid  in  ship  procurement  is 
not  a  new  concept.     As  early  as   1958  an  ashore  navigation  center  was 
constructed  by  Sperry  Systems  Management  Division  in  support  of  the 
SSBN  598  class  of  submarine  construction.      Test  sites   employing 
specific  equipments  are  presently  available  at  additional  locations 
such  as  Mare  Island.     However,    the  PFCSLBTS  is  the  first  site  con- 
structed which  is  dedicated  to  a  single  class  of  ship. 

Recent  experience  with  the  DLGN-36  class  procurement  has  re- 
inforced the  need  for  a  dedicated  test  site.     A  review  group  under  the 
banner  of  the  CNM  recently  (14  March  1974)  completed  an  audit  of  the 
problems  found  in  the  DLGN-36  combat  system.      Throughout  the  report 
it  is   repeatedly  pointed  out  that  the  lack  of  a  LBTS  seriously  hampered 
and  delayed  successful  combat  system  integration. 
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For  the  DLGN-36  procurement  and  others   (DD-963,    DE-1052  etc.  ), 
land-based  testing  was  conducted  on  a  fragmented  subsystem  basis. 
Some  subsystems  were  partially  tested  at  Mare  Island  for  instance, 
and  others  at  Dam  Neck.     In  neither  case  was  there  a  "full  up"  capa- 
bility for  testing  the  complete  combat  system.      The  first  time  the 
major  subsystems  were  all  brought  together  at  one  place  was  aboard 
ship.     Since  there  was  no  land-based  test  site,    the  full  up  system 
problems  were  not  encountered  until  the  advanced  stages  of  shipboard 
testing.     Such  is  not  the  case  for  the  PF  procurement  program. 

B.      PATROL  FRIGATE  LAND-BASED  TEST  SITE  PHYSICAL  PLANT 
The  LBTS  is  located  at  MacArthur  Airport,    Islip,    New  York.     Its 
primary  initial  function  is  to  serve  as  a  facility  for  the  integration  of 
the  combat  system  equipment  and  for  training  a  U.    S.    Navy  crew  to 
demonstrate  the  system  prior  to  the  PF  production  decision.     It  is 
anticipated  that,    following  the  production  decision,    the  LBTS  will  be 
used  for  crew  training,    shipyard  personnel  training,    computer  program 
verification,    change  verification,    test  program  checkout,    etc.      The 
LBTS  will  also  be  used  for  design  and  test  of  system  modifications. 

The  building  which  houses  the  LBTS  contains  the  physical  mock-up 
(PMU)  compartments  for  the  PFCS,    the  test  facility  for  the  Mk  92   FCS, 
and  facilities  for  support  of  these  activities.     In  addition,    the  LBTS 
contains  the  offices  and  classrooms   required  to  support  this  plan. 
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Appendix  B  depicts  the  location  of  the  test  site  in  relation  to  MacArthur 
Field,    a  layout  of  the  test  site,    a  plan  view  of  the  CIC  and  sonar  control 
room,    the  CIC,    IFF  and  sonar  equipment  room,    the  Mk-92  equipment 
room,    the  STIR  (Separate  track  and  illuminate  radar)  equipment  room, 
the  operations  center  (OCC),    the  program  checkout  area  (PCA),    and 
the  Mk-92  test  site.     Radar  antennas  are  physically  located  on  the 
northern  side  of  the  test  center  in  the  approximate  configuration  that 
they  will  appear  onboard  ship  with  the  exception  of  height  which  is 
limited  by  FAA  regulations. 

"The  physical  mock-up  compartments  will  contain  actual  operating - 
equipment  as  well  as  simulators  and  mock-ups,    and  will  represent  the 
precise  configuration  of  the  ships  compartments.      The  CIC  physical 
mock-up  contains  functional  communications  circuits  to  enable  the 
simulation  of  tactical  operations.      The  LBTS  also  employs  functional  _ 
SPS-49,    SPS-55  and  Mk  92  Mod  2  CAS  (combined  antenna  system)  and 
STIR  radars.     It  is  thus  capable  of  detecting  and  tracking  actual 
aircraft.     No  capability  exists,    however,    for  actual  subsurface  targets 
and  only  a  limited  capability  exists  for  surface  targets.     Ships  motion, 
as  well  as  air,    surface,    and  subsurface  targets  can  be  simulated  by  an 
OCC  Console  and  inserted  into  the  command  and  control  element  of 
the  combat  system.      Based  on  these  simulated  data  inputs,    the  combat 
system  will  develop  CIC  console  scope  presentations  and  data  readouts 
depicting  test  or  tactical  operation  scenarios.      Operator  actions  will 
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result  in  selection  and  ordering  of  simulated  weapons.      During  the 
performance  of  the  exercises,    CIC  operator  actions  and  displayed 
console  data  can  be  monitored,    recorded,    and  evaluated  through  use 
of  closed  circuit  TV,    PPI  displays,    data  readouts,    and  recording 
equipment  on  the  OCC  console. 

The  PMU  is  realistic  to  the  point  of  being  deceptive,    and  only  the 
lack  of  ships  motion  and  the  smell  of  salt  air  detract  from  almost 
complete  authenticity. 

C.      USES  ENVISIONED  FOR  THE  CSLBTS 

1.  DSARC  III  Demonstration 

As  previously  stated,    the  primary  short-range  use  of  the 
LBTS  is  to  prepare  a  presentation  for  the  production  decision  at 
DSARC  III.     In  fact,    there  are  those  close  to  the  PF  project  who 
believe   that  the  production  decision  hinges  almost  completely  on  how 
well  the  CSLBTS  reacts  to  the  scenarios  during  the  presentation.     In 
preparation  for  this  demonstration,    a  Navy  crew  is  presently  under- 
going training  under  Sperry  guidance  to  gain  the  knowledge  and  skills 
necessary  to  demonstrate  satisfactory  operation  of  the  PFCS  during 
the  IOT&E  demonstration.     A  limited  number  of  COMOPTEVFOR 
personnel  will  also  attend  this  training. 

2.  Training 

This  training  includes  the  following: 
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-The  characteristics  of  the  PFCS  and  the  LBTS,    to  include 
system  capabilities  and  limitations,    mission,    combat  system 
elements,    subsystem  elements  and  manning. 

-Combat  system  functional  data  flow,    including  the  signal 
characteristics  and  time  relationships. 

-The  IOT&E  tests  to  be  performed  on  the  PFCS  including 
threats,    test  method  and  test  procedures. 

-The  theory  needed  to  support  and  understand  the  performance 
of  operational  procedures,    for  detection,    identification, 
evaluation,    weapon  assignment  and  weapon  control  under 
various  threat  conditions. 

-The  theory  needed  to  perform  operational  trouble  analysis 
and  fault  isolation  to  the  subsystem/equipment  level  by 
employment  of  system  operational  tests  and  programmed 
operational  functional  appraisals  and  by  logical  analysis 
using  maintenance  documentation. 

-Documentation  and  procedures  to  support  system  level 
organizational  maintenance. 

-Ability  to  perform  normal  and  casualty  operations  on  an 
assigned  station  with  an  acceptable  probability  of  successful 
target  engagement,    for  each  of  the  following  scenarios: 

--Power  up,    Condition  3. 

--Work  station  checkout,    Conditions  3  and   1. 


.  - 
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--Single  aircraft  attack,    Conditions  3  and  1. 
--Single  surface/near  surface  attack,    Conditions  3  and  1. 
--Single  subsurface  attack,    Conditions  3  and  1. 
--Multiple  threat,    Conditions  3  and  1. 
--Casualty  mode,    Conditions  3  and  1. 
-Ability  to  perform  trouble  analysis  and  fault  isolation  of 
PFCS  faults  to  the  equipment  level,    using  diagnostic  computer 
software  and  maintenance  manuals. 
-Ability  to  perform  system  level  organizational  maintenance. 
-Ability  to  perform  the  IOT&E,    with  acceptable  accuracy 
and  within  specified  time  limits,    using  specified  methods 
and  procedures. 
The  objectives  of  the   lead  shipbuilder  course  is  to  familiarize 
shipbuilder  personnel  with  the  PFCS  and  to  provide  them  with  the  unique 
skills  and  knowledge  required  to  install,    integrate,    and  check  out  the 
combat  system  on  the  Patrol  Frigate.      The  follow  shipbuilder  course 
will  be  the  same  as  the  lead  shipbuilder  course  except  for  modifications 
to  accomplish  curriculum  improvement,    as  necessary  to  reflect  ship- 
board configurations,    or  as   required  by  the  follow  shipbuilders. 

As  a  training  vehicle,    the  LBTS  is  limited  by  the  following 
considerations : 

-Until  U.    S.    Navy  certification  15  December  1974,    priority  of 
use  of  facilities  and  equipment  is  assigned  to  engineering 
development,    system  integration,    and  testing. 
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-After  15  December  1974  and  until  IOT&E  is  completed, 
priority  of  use  will  be  assigned  by  the  U.   S.    Navy  OIC  of  the 
LBTS. 

-The  LBTS  equipment  complement  does  not  include  all  ship- 
board CIC  equipment.     Simulation  is  provided  in  lieu  of  major 
system  inputs  such  as  sonar,    ECM,    ships  motion,    and  weapons 
dynamics.      Mock-ups  are  provided  for  major  equipment  found 
in  the  shipboard  configuration  but  not  essential  to  combat 
system  IOT&E. 

-Unscheduled  (corrective)  maintenance  will  be  performed  by- 
Sperry  engineering  and  contractor  technical  support  rather 
than  U.    S.    Navy  personnel. 

-Deliberately  faulted  modules  will  not  be  available  or  authorized 
for  use  in  the  system  prior  to  or  during  the  IOT&E. 
3.       General  areas 

The  specifics  of  the  majority  of  the  tests  to  be  conducted  at 
the  LBTS  are  classified.  However,  the  LBTS  will  be  used  as  a  test 
bed  to  address  the  following  type  of  questions: 

-Can  the  PF  develop  attack  data  on  submarines  by  the  use  of 
active  sonar  while  maintaining  speeds  up  to  X  knots? 

-Can  the  PF  decoy  passive  acoustic  homing  torpedoes? 

-Is  the  PF  capable  of  employing  ASW  weapons  to  the  effective 
range  of  installed  subsurface  sensors? 
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-Is  the  PF  capable  of  controlling  HS/VS/VP  aircraft? 
-Is  the  PF  capable  of  detecting  airborne  threats  by  use  of 

radar? 
-Can  the  PF  detect  and  evaluate  electromagnetic  radiation? 
-Is  the  PF  capable  of  tracking  and  engaging  fighter-bomber 

type  aircraft? 
-Can  the  PF  track  and  engage  air,    surface  or  sub-surface 

launched  anti- shipping  missiles? 
-Is  the  PF  capable  of  engaging  surface  targets  to  over-the- 

horizon  ranges? 
-Is  the  crew  capable  of  performing  all  combat  systems 

operations? 
-Is  the  crew  capable  of  performing  all  combat  system 

maintenance  tasks? 
4.       Documentation 

One  of  the  most  difficult  and  expensive  areas  to  manage  in 
the  procurement  of  a  new  combat  system  is  documentation,    especially 
at  the  system  level.      The  PFCSLBTS  will  aid  documentation  as  follows: 
-Generation  of  publications  engineering  data  and  text. 
-Verification  of  system  documentation  for  builder  and  fleet 

use 
-System  Documentation 

--Technical  manuals   (system  and  equipment) 
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--Standard  operating  procedures 
--SMP's  (System  maintenance  procedures) 
--PMS  (Planned  maintenance  system) 
--System  computer  program  and  documentation 
--System  and  equipment  diagnostic  computer  programs 
-Data  flow  and  one-function  diagrams. 

5.  Logistic  support 

Logistic  support  for  equipments  in  combat  systems  are  often 
based  on  engineering  data  and  test  bench  operation  by  equipment  manu- 
facturers.     For  the  PF  there  is  a  Failure  Data  Collection,    Analysis, 
and  Corrective  Action  Plan  which  is  designed  for  the  unique  require- 
ments of  the  Land  Based  Test  Site  while  permitting  the  acceptance  of 
data  from  vendor  facilities,    shipbuilders,    government  sources  and 
eventual  fleet  operations.      This  plan  incorporates  failure  data  collec-  . 
tion,     failure  analysis,    corrective  action,    and  failure  reporting.      Thus 
it  may  be  viewed  as  a  dual-loop  system  whereby  better  logistic  support 
requirements  may  be  determined  initially  based  on  LBTS  data  and  in 
the  long  run  on  a  system  whereby  continuing  assessment  of  support 
requirements  based  on  fleet  data  may  be  made. 

6.  Long  term  uses 

Sperry's  experience  with  the  SSBN  598  class  ashore-navigation 
center  seems  to  indicate  that  in  the  short  run  the  configuration  of  a 
given  LBTS  closely  duplicates  the  class  of  ship  for  which  it  was  built. 
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As  time  goes  on  and  other  classes  of  ships  come  into  being,    there  is  a 
tendency  to  modify  existing  LBTS  to  accommodate  ashore  testing  of 
new  systems.     In  this  regard,    the  LBTS  is  a  vehicle  to  groom  tactical 
equipment,    and  verify  installation  test  procedures  and  system  computer 
programs.     It  supports  system  improvement  studies,    development  of 
new  system  concepts,    and  analysis  of  advanced  system  computer  pro- 
gram concepts.     In  development  areas,    the  LBTS  may  be  used  to  test 
and  evaluate  new  operational  techniques,    system- equipment  changes, 
define  change  requirements,    evaluate  proposed  changes,   and  to  verify 
shipalt/ordalt  change  documentation  and  accuracy. 
7.       Land-based  Test  Site  Users 

While  the  total  spectrum  of  users  of  the  PFCSLBTS  can  only 
be  determined  after  the  fact,    it  is  envisioned  that  many  types  of  activi- 
ties will  use  the  LBTS  to  advantage.      Users  envisioned  at  the  present 
time  include  procurement  decision  makers,    system  designers,    system 
analysts,    system  programmers,    equipment  contractors  and  sub- 
contractors,   publication  agencies,    integrated  logistic  support  personnel, 
fleet  personnel,    field  engineers,    and  shipbuilders.     In  addition,    relia- 
bility managers,    those  involved  in  quality  assurance,    and  politicians 
are  sure  to  make  use  of  the  facilities  available  at  the  LBTS. 

D.      GOALS  OF  THE  PATROL  FRIGATE  LAND-BASED  TEST  SITE 

The  initial  goal  of  the  PFCSLBTS  is  to  adequately  demonstrate  the 
viability  of  the  proposed  combat  system  to  the  DSARC  at  the  production 
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decision  and  thus  support  fleet  requirements  in  the  1980's  and  90's. 
The  other  side  of  this   goal,    so  to  speak,   has  a  negative  connotation. 
To  wit,    if  the  system  is  not  viable,    it  will  not  be  procured.     Within  the 
initial  goal,    there  are  many  subgoals  present.     These  subgoals  include 
saving  money  and  time,    and  to  ensure  design  soundness,    requirements, 
and  equipment  system  compatibility.     Additional  goals  include  the  veri- 
fication of  publications  such  as  system  operating  and  maintenance  pro- 
cedures,   and  system  computer  programs.      Finally,    it  is  a  goal  of  the 
LBTS  to  provide  the  means  to  evaluate  equipment  performance  and  to 
solve  system  technical  problems  before  they  develop  in  the  fleet  and 
when  they  do,    to  provide  a  system  to  investigate  and  resolve  them 
quickly. 

E.      THE  LAND-BASED  TEST  SITE  CONTRACT 

Sperry  Systems  Management,    an  operating  unit  of  Sperry  Division, 
Sperry  Rand  Corporation,    presently  holds  a  $12  million  dollar  contract 
with  Naval  Ship  Systems   Command  (NAVSHIPS)  covering  their  effort  in 
support  of  the  PFCSLBTS.     Some  highlights  of  that  contract  (#N0024- 
73-C-1089)  follow. 

1.       Test  and  Evaluation 

Sperry  Management  Division  is   required  to  design,    erect,    and 
operate  the  land-based  test  site.      The  LBTS  is  to  be  used  as  a  facility 
for  integration  and  validation  of  the  PF  combat  system  prior  to  the 
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PF  production  decision.      The  Mk  92  MOD  2  fire  control  system  being 
produced  by  the  contractor  under  separate  contract  is  the  principal 
element  of  the  combat  system.      The  CSLBTS  will  be  used  for  acceptance 
evaluation  of  the  Mk  92   FCS  and  its  subsequent  integration  with  the  GFE 
elements,    simulators,    switchboards  and  ancillary  equipments  of  the 
PFCS  for  test  validation  and  certification  of  the  PFCS. 

2.  Construction  of  the  LBTS 

The  LBTS  is  to  include  training  facilities  and  the  contractor 
shall  define  and  provide  system  simulators  and  programs  required, 
prepare  equipment  program  specification,    statements  of  work  and  all 
other  contractual  documents  necessary  for  procurement  of  the  LBTS 
construction.      The  contractor  will  check  out  simulation  equipment  and 
programming  to  ensure  adequate  performance  to  tactical  equipment's 
specifications . 

3.  Support  activities 

Under  direction  of  the  Navy  OIC  of  the  LBTS,    the  contractor 
is  to  provide  a  technical  director  to  operate  and  maintain  the  LBTS 
and  PCA,    as  a  development  demonstration  proofing  site  for  the  PFCS. 
The  contractor  shall  provide  all  facilities  required  by  the  OIC  for  the 
LBTS  to  include  provision  for  and  maintenance  of  the  office  space 
required  by  the  OIC,    provision  for  reproduction  services,    provision 
for  clerical  and  custodial  services,    provision  and  maintenance  of 
appropriate  security  services  and  precautions. 
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4.  Repair  parts 

The  contractor  will  be  provided  spare  and  repair  parts  as 
determined  by  the  government  for  government  furnished  equipment. 

5.  PFCS  Integration  and  Test 

The  contractor  shall  integrate  the  Mk  92  MOD  2   FCS  produced 
by  the  contractor  under  separate  contract  with  government  furnished 
computers,    display,    console  sensor  and  other  elements  of  the  combat 
system  at  the  LBTS.      The  contractor  is  required  to  verify  the  integra- 
tion of  the  combat  system  at  the  LBTS  in  accordance  with  a  test  plan 
to  be  provided  by  the  contractor  and  approved  by  the  Navy.      The  test 
plan  shall  be  developed  from  an  integrated  test  outline  furnished  by 
the  Navy  and  shall  be  directed  toward  the  verifying  of  combat  system 
performance  including  the  PFCS  performance  specification  provided 
by  the  Navy.      The  contractor  shall  provide  all  test  maintenance,    com- 
puter support  personnel,    and  such  simulation  programs  as  are  required 
for  the  performance  of  this  task.      The  contractor  shall  also  collect, 
compile  and  analyze  all  data  and  prepare  reports  as   required.      The 
contractor  shall  develop  appropriate  test  plans  for  performance  of  the 
IOT&E  demonstration  and  submit  the  test  plans  for  SHAPM  approval. 

6.  LBTS  Operation  and  Demonstration  Support 

The  contractor  shall  support  Navy  personnel  as   required  in 
performing  the  IOT&E  demonstration  at  the  LBTS.      The  contractor 
shall  support  other  demonstrations  and  evaluations  conducted  at  the 
LBTS  under  direction  of  the  OIC.      The  contractor  shall  staff  and 
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operate  the  LBTS  facility  under  supervision  of  the  OIC  after  completion 
of  the  initial  PFCS  integration,    testing,    certification  and  IOT&E  demon- 
stration as  a  test   development  and  training  center.      The  facility  will  be 
utilized  to  support  the  PFCS  in  all  areas   requiring  the  use  of  an  opera- 
tional test  bed.     Area  use  anticipated  for  the  LBTS  includes: 

-Documentation  development  and  verification 

-Combat  system  crew  training  for  follow  ships 

-Special  demonstrations 

-Investigation  of  combat  system  changes 

-Prove  out  of  ordalts  and  shipalts 

-Fleet  combat  system  problem  investigation 

-Integration  of  growth  items 

-Shipyard  support. 
7.       Test  Procedures 

Utilizing  the  integrated  test  outline  and  the  combat  system 
performance  specification  prepared  by  the  Navy  and  furnished  to  the 
contractor,    the  contractor  shall  prepare  and  maintain  a  current  test 
and  evaluation  plan,    and  a  set  of  test  procedures  for  use  in  the  opera- 
tion,   test,    evaluation  and  demonstration  of  the  PFCS  at  the  LBTS  and 
eventual  use  on  the  lead  ship.      The  test  plan  and  procedures  shall  cover 
the  following: 

-Incoming  inspection  of  GFE 

-Unit  test  of  GFE 
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-Subsystem  testing  of  GFE 

-Integration  of  the  Mk  92  MOD  2   FCS  produced  by  the  contractor 
under  another  contract  with  material  furnished  under  this  con- 
tract and  the  GFE  in  the  LBTS. 

-Combat  system  test  demonstration  at  the  LBTS. 
8.      SHAPM  Support 

The  contractor  is   required  to  support  the  SHAPM  in  the 
development,    test  and  evaluation  of  the  PFCS  by  maintaining  control  of 
the  configuration  of  the  PFCS  at  the  LBTS  until  satisfactory  completion 
of  the  IOT&E  demonstration.      The  contractor  is  required  to  furnish 
technical  support  to  the  lead  ship  builder  in  the  design  of  the  PFCS 
compartments.     He  is  required  to  assist  the  SHAPM  in  the  resolution 
of  problems  directly  associated  with  the  PFCS,    specifically  in  the  area 
of  integration  analysis,    documentation,    ship  test  planning  and  develop- 
ment,   combat  system  reliability  and  maintainability  and  availability 
analysis  and  evaluation. 

The  contract  as  outlined  above  expires  in  March  of  1975  after 
the  scheduled  DSARC  III  presentation.     It  is  conjectured  that  the  con- 
tract will  be  extended  on  at  least  a  yearly  basis  in  support  areas. 
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VII.     SOME  PFCSLBTS  CONCEPTS  AMPLIFIED 

A.  GENERAL 

Contracts,    test  plans,    operating  procedures  and  Congressional 
testimony  in  general  address        the  theoretical  side  of  procurement; 
what  is  planned  or  ought  to  be  rather  than  what  is.     Since  the  Patrol 
Frigate  LBTS  can  be  viewed  as  a  prototype  of  land-based  test  sites 
for  future  ship  procurements,    it  is  of  interest  to  get  behind  the  scenes 
to  the  grass  roots  level  of  test  site  concepts.     Information  presented 
in  the  following  section  is  an  attempt  to  document  the  beliefs,    gut 
feelings  and  thoughts  of  a  myriad  of  individuals  and  commands  contacted 
and/or  visited  during  research  for  this  thesis.      The  author  and  the 
author  alone  is  responsible  for  the  inferences  contained  herein. 

B.  A  COLLAGE  OF  QUESTIONS  INCLUDING  A  FEW  ANSWERS 

1.       What  working  relationships  have  evolved  between  the  LBTS 
OIC  and  SHAPM  (PMS-399),    Sperry  NAVPRO  etc.    in  the  three  month 
period  the  LBTS  has  been  manned  with  Navy  personnel? 

The  OIC's  marching  orders  at  this  early  point  in  the  LBTS  life 
are  essentially  to  get  the  job  done.     Since  the  concept  of  what  the  job  is 
is  evolving  in  parallel    to  the  LBTS,    tasks  have  not  been  formalized 
to  any  great  extent,    nor  should  they  be.     At  the  present  time  the  OIC 
has  broad  duties  and  freedom  in  his  actions.     He  has  submitted    a 
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one-page  recommendation  as  to  what  his  duties  should  entail,    and 
PMS-399  is  in  the  process  of  integrating  this  input  into  a  more  formal 
job  description. 

At  the  present  time  the  OIC  reports  to  NAVSHIPS  as  an  OIC, 
and  to  PMS  399  as  PMS  399.  16,    OIC  of  the  PFCSLBTS.     The  NAVSHIP 
reporting  chain  enables  the  OIC  to  receive  support  such  as  being  on  the 
SNDL  (Standard  Navy  Distribution  List),    have  his  own  UIC  (User  Identi- 
fication Code),    and  enables  him  to  receive  outside  assistance  that  could 
be  cumbersome  to  receive  via  the  SHAPM  chain.      The  day-to-day  chain 
of  command  is  through  PMS-399  and  it  is  through  this  chain  that  fitness, 
reports  will  be  generated. 

The  OIC  is  administratively  supported  by  the  NAVPRO  office 
at  Sperry  Great  Neck,    some  50  odd  miles  from  the  LBTS.      This  sup- 
port presently  takes  the  form  of  Naval  communication  support,    NAV- 
PRO PF/MK  92-2  contract  inputs,    and  various  other  administrative 
support  items.      The  NAVPRO  is  administering  the  Mk  92-2  contract 
with  Sperry,    but  the  PFCSLBTS  contract  is  being  administered  by 
contracting  personnel  on  the  SHAPM's  staff  and  by  NAVSHIPS  025. 

The  OIC's  relationships  with  outside  activities  such  as  NAVSEC, 
OPTEVFOR  etc.  ,    generally  falls  in  the  area  of  assisting  Sperry  Man- 
agement Division  in  the  prosecution  of  their  contract.     Informally,    the 
OIC  assists  whomever  he  can.      For  instance,    assistance  is  being  pro- 
vided to  NAVORD  personnel  in  tracking  NAVORD's  portion  of  the  Mk  92 
procurement  contract. 
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The  OIC  is  on  a  razor's  edge  between  those  who  recognize 
a  need  in  the  fleet  for  a  ship  weapons  system  such  as  the  PF  and  those 
who  are  test  oriented,    and  at  the  extreme  test  for  testing's  sake.     The 
former  group  are  equipment/production  oriented  and  include  both  pro- 
ducers of  combat  system  equipment  such  as  Sperry  and  operational 
personnel  who  perceive  that  the  system,    albeit  with  short  falls,    is 
better  than  what  is  presently  in  the  fleet.      The  latter  group,    the  "testers", 
many  times  have  to  devote  much  time  supporting  the  rationale  of  T&E 
reports  before  GAO,    Congress,   and  the  Office  of  the  Secretary  of 
Defense  to  prevent  misinterpretation.      The  reporting  of  T&E  results 
culminates  the  T&E  cycle  and  forms  the  basis  for  various  decisions 
concerning  the  future  status  of  the  weapon  system.      For  example, 
IOT&E  reports  are  used  by  Congress  as  set  forth  in  PL  92-156,    and 
by  the  DSARC  as  an  input  for  the  major  production  decision.      They  are  , 
also  used  by  COMOPTEVFOR  to  support  the  suitability  of  a  proposed 
system  to  meet  fleet  needs  and  operability  requirements. 

In  view  of  the  importance  of  T&E,    different  agencies  have 
different  emphases  in  the  conduct  of  the  T&E.     A  grouping  of  the 
thrust  of  T&E  needs  could  be  the  cost-effectiveness  group  versus 
operators  and  producers.     Within  the  operator /equipment  side,    the 
LBTS  OIC  as  stated  in  Chapter  V  has  authority  to  make  changes. 
Once  the  test  group  presents  the  OIC  with  a  test  plan  for  IOT&E  and 
DSARC  III  demonstration,    the  OIC  becomes  an  instrument  of  the  test 
group  and  his  ability  to  alter  plans  is  severely  limited. 
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2.       What  authority  does  the  OIC  have  within  the  boundaries  of  the 
contract?     What  is  his  relationship  to  testing  and  test  and  evaluation 
activities? 

Authority  to  make  changes  in  contractual  obligations  is 
severely  limited,    as  it  should  be,    in  that  the  OIC  does  not  have  the 
resources  to  carry  out  complex  changes.     The  OIC  can  make  changes 
that  involve  no  change  in  the  scope  of  the  contract  and  upon  which  Navy- 
direction  is   required.      PMS-399  maintains  centralized  authority  to 
make  contract  changes. 

Since  the  effort  required  to  derive  proper  tests  and  test  pro- 
cedures was  underway  long  before  the  OIC  reported  to  the  LBTS  in 
January  1974,    the  OIC  had  no  input  into  test  development.     He  has 
however,    become  deeply  involved  in  the  evaluation  of  the  CSOT  (Combat 
System  Operability  Test),    which  will  be  conducted  by  Navy  personnel 
both  at  the  LBTS  and  throughout  the  life  cycle  of  projected  ships. 

The  OIC  views  testing  at  the  LBTS  as  a  three-phase  effort 
as  follows: 

-Tests  leading  up  to  Navy  acceptance/certification  of  the 
LBTS  in  December   1974.     Sperry  is  conducting  these  tests  based  on 
outlines  supplied  by  the  Navy.      The  purpose  of  these  tests  is  to  demon- 
strate a  total  integrated  combat  system  to  the  Navy  as  witnessed  by 
the  OIC. 
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-IOT&E  tests.      The  purpose  of  these  tests  is  a  demonstration 
by  the  Navy  to  DOD  and  DSARC  III  of  the  operational  characteristics  of 
the  combat  system.     The  tests  will  include  scenarios,    simulated  air- 
craft,   and  actual  aircraft  engagement.      The  OIC  and  Navy  personnel 
will  conduct  these  tests.     The  tests  will  be  developed  by  PMS-399, 
APL  (Applied  Physics  Laboratory),    OPTEVFOR,    and  other  agencies. 

-Prove  out  and  conduct  lead  ship  tests.      These  tests,    from 
the  Total  Ship  Test  Program  (TSTP),    will  be  conducted  to  verify  that 
the  lead  ship  can  indeed  be  checked  out  and  delivered  in  an  up  condition. 
These  tests  will  be  conducted  by  the  OIC  and  Navy  personnel. 

At  various  stages  in  the  test  phasing  outlined  above,    different 
agencies  will  be  involved  in  the  actual  data  collection,    data  reduction, 
and  analysis.      These  responsibilities  are  presently  being  considered. 

3.  What  formal  and  informal  relationship  exists  between  the  OIC 
of  the  LBTS  and  other  NAVPRO  personnel? 

There  is  no  formal  relationship  such  as  one  would  find  on  an 
organizational  chart.      The  OIC  is  working  with  the  NAVPRO  toward 
acceptance  of  the  PFCS  by  the  U.    S.    Government.      There  is  presently 
under  preparation  an  agreement  between  NAVSHIPS  and  the  Sperry 
NAVPRO  whereby  the  NAVPRO  would  be  tasked  to  assist  the  OIC  in 
all  technical  matters  relating  to  the  LBTS. 

4.  What  is  the  status  of  follow-on  contracts  with  Sperry  Manage- 
ment for  such  things  as  life  cycle  support  of  the  program? 
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Review  of  the  present  contract  and  the  substance  of  future 
contracts  will  come  under  scrutiny  in  the  fall  of  1974.     Unlike  most 
contracts,    it  is  envisioned  that  the  contract  for  life  cycle  support  of 
the  program  will  be  a  multiple  year  contract. 

The  LBTS  is  viewed  as  a  ship  permanently  home  ported  in 
Sperry's  building  at  MacArthur  which  the  Navy  runs.     Sperry  and  other 
industrial  support  is  viewed  much  like  what  afloat  units  receive  from 
tenders  and  ship  yards. 

5.       How  will  personnel  be  cycled  through  the  LBTS  for  training? 

A  solid  long  range  plan  has  not  been  developed  at  this  point  in 
time.     Up  through  IOT&E,    there  will  be  a  crew  of  about  22  men  includ- 
ing three  officers  and  one  Wave  resident  at  the  LBTS.      These  people 
will  constitute  the  Navy  crew  for  the  DSARC  III  demonstration,    and 
then  will  eventually  be  assigned  to  the  lead  and  follow  ships.      In  parallel, 
additional  people  will  cycle  through  the  LBTS  for  training. 

About  six  to  twelve  sailors  will  be  permanent  residents  of 
the  LBTS  under  one  officer  in  the  grade  of  LCDR/CDR.      They  will 
operate  and  maintain  the  site  and  cycle  through  changes  etc.   as 
required. 

It  is  planned  to  instruct  civilian  workers  from  the  three 
builders  who  win  the  PF  procurement  contract  in  all  aspects  of  the 
installation  and  tests   required  by  the  PFCS.      These  individuals  in 
turn  will  supervise  installation  and  test  of  the  PFCS  at  their  respective 
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building  sites  and  instruct  Navy  precommissioning  crews.  These 
crews  in  turn  will  form  the  nucleus  of  the  crew  that  takes  the  ship 
through  trials  and  eventually  to  the  fleet. 

6.  What  impact  would  stretching  out  of  the  production  decision 
at  DSARC  III  have  on  operation  of  the  LBTS? 

The  LBTS  was  primarily  built  as  a  vehicle  to  demonstrate 
the  PFCS  at  DSARC  III,    and  in  the  long  term,    to  support  the  life 
cycle  of  the  ship.     In  this  regard,    at  least  in  the  short  run,    activity 
at  the  LBTS  is  directed  toward  the  DSARC  III  presentation.      These 
activities  of  themselves  should  be  the  cause  for  any  delay  of  the  produc- 
tion decision  at  DSARC  III,    rather  than  DSARC  III   causing  the  stretch- 
ing out.     Other  than  a  complete  cancellation  of  the  PF  program  then, 
the  LBTS  is  here  to  stay. 

7.  What  is  considered  to  be  the  "salvage  value"  of  the  test  site?    , 
Assuming  that  the  program  receives  a  green  light  at  the  pro- 
duction decision,    a  realistic  salvage  value  cannot  be  assigned  to  the 
site  at  this  time.      That  can  only  be  really  determined  when  the  site 

is  closed  down. 

The  major  purpose  of  the  LBTS  is  to  determine  if  it  is  feasible 
to  do  what  you  want  to  do.     In  effect  money  is  being  spent  now  so  that 
fifty  times  as  much  isn't  required  in  the  future  to  obtain  the  same 
results,    whether  good  or  bad.      The  $12  million  or  so  invested  in  the 
LBTS  is  certainly  a  small  cost  in  comparison  with  a  projected  cost 
of  over  $3  billion  for  the  fifty  ship  buy. 
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8.  How  closely  does  the  LBTS  duplicate  both  human  and  machine 
operations  of  the  combat  system  aboard  ship? 

It  is  the  intent  to  duplicate  as  closely  as  possible  shipboard 
conditions.      Equipment  limitations  and  the  use  of  simulation  was 
addressed  in  Chapter  5.      From  the  human  aspect,    the  LBTS  is  almost 
identical  to  a  shipboard  installation  down  to  fire  extinguishers  on  the 
bulkheads  and  Playmates  posted  in  switchboard  doors. 

Environmental  factors  preclude  bringing  jet  aircraft  against  the 
site  at  high  speeds  and  low  altitudes,    and  simulators  will  have  to  be 
relied  upon  heavily  for  such  tests.     However,    real  life  tends  to  be  not 
as  confusing,    mixed  up,    nor  saturated  as  the  situations  presented  via 
simulations.      In  simulations,    the  kitchen  sink  can  be  thrown  in  but 
realistically,    there  are  only  so  many  aircraft  that  can  come  at  you, 
and  they  are  as  confused  as  you  are  as  to  what  the  situation  is. 

9.  What  will  be  the  training  status  of  the  crew  used  for  the 
DSARC  III  demonstration? 

For  the  PFCS,    this  question  lends  itself  naturally  to  another 
one.     How  can  a  combat  system  be  tested  under  ideal  conditions  when 
the  training  period  leading  up  to  the  test  period  is  minimal?     Are  you 
testing  the  combat  system  or  are  you  testing  the  training  of  the  demon- 
stration crew? 

At  the  LBTS,    the  Navy  crew  will  have  approximately  one  month 
"hands  on"  training  prior  to  the  start  of  IOT&E.     In  this  short  time 
frame,    they  will  be  required  to  reach  a  level  of  competency  often  not 
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reached  by   fleet  units  during  a  six  month  buildup  and  several  weeks  of 
refresher  training. 

Additionally,    there  exists  an  educational/technical  gap  between 
Navy  and  contractor  T&E  personnel.      The  contractor  is  represented  by 
well-paid,    highly  experienced,    and  technically  competent  engineers  and 
technical  support  personnel,    which  gives  him  the  edge  in  structuring 
the  T&E  effort  to  his  contractual  advantage. 

10.       How  is  approval  for  service  use  as  a  result  of  DSARC  III,    and 
the  Board  of  Inspection  and  Survey  (BIS)  acceptance  for  service  use 
from  acceptance  trials  viewed  by  LBTS  personnel? 

Sperry  and  LBTS  Navy  personnel  understand  that  they  are 
assembling  a  combat  system  to  accomplish  the  mission  required.      Their 
effort  is  directed  toward  making  the  beast  work.      The  BIS  in  its  accept- 
ance trials  tests  not  only  how  well  the  combat  system  performs  its 
assigned  tasks,    but  brings  in  points  of  contention  such  as  whether  a 
certain  light  should  be  red  or  amber,    are  phone  jacks  located  in  acces- 
sible places   (even  though  they  may  have  been  located  in  accordance  with 
MILSPECS)f    and  other  such  considerations. 

Admiral  Zumwalt,    testifying  before  a  hearing  on  Cost  Escala- 
tion in  Defense  Procurement  Contracts  in  1973,    stated  that  he  had 
command  of  a  destroyer  in  1957  which  was  one  of  100  consecutive 
destroyers  found  to  be  unsatisfactory  by  the  BIS.      "Yet  my  ship  won 
the  efficiency  pennant  in  every  department  of  the  ship  and  the  battle 
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efficiency  pennant  and  yet  if  you  read  the  INSURV  report,    we  couldn't 
go  to  war.  " 

The  best  way  to  handle  deficiencies  is  to  make  sure  they  never 
arise  in  the  first  place  because  of  design  or  fabrication  shortfalls.      Thus 
it  appears  that  there  should  be  a  closer  relationship  between  BIS  per- 
sonnel charged  with  acceptance  of  systems  and  personnel  involved  in 
presenting  a  system  to  the  DSARC  for  service  approval. 

11.  How  flexible  is  the  site  for  expansion  of  PF  projects?.      For 
alteration  to  similar  ship  systems? 

The  PMU  is  a  physical  model  of  the  actual  PF  shipboard 
spaces.     As  such  it  is  a  very  useful  tool  to  aid  in  determining  the  space 
allocation  requirements  for  growth  items.     The  LBTS  itself  is   space 
limited  to  a  certain  degree  within  the  existing  building  if  it  is  desired 
to  build  PMU's  of  other  classes  of  ships.     However  space  does  exist,, 
whereby  equipment  could  be  placed  on  pallets  and  connected  to  the 
PCA  to  conduct  various   equipment /computer  interface  tests.     Real 
estate  wise,    grounds  exist  to  extend  the  existing  building  by  a  con- 
siderable amount. 

12.  What  systems  will  be  taken  from  the  LBTS  to  Bath  for  instal- 
lation aboard  the  lead  ship?     Will  specific  subsystems  be  tested/ 
integrated  at  the  LBTS  prior  to  being  sent  to  the  shipbuilders? 

There  is  a  possibility  that  all  the  equipment  comprising  the 
PMU  installation  will  be  sent  to  Bath  for  installation  aboard  the  lead 
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ship.     In  this  manner,    shipboard  installation  and  check  out  would  have 
as  its  basis  a  completely  checked  out  combat  system.      Presently  under 
consideration  is  what  specific  subsystems,    if  any,    should  be  cycled 
through  the  LBTS  prior  to  shipboard  installation.      Even  a  partial 
cycling  of  critical  units  through  the  LBTS  would  improve  configuration 
control,    enhance  check-out  of  the  equipment,    and  provide  an  opportunity 
for  LBTS  personnel  to  bring  equipment  up  to  the  latest  change  status 
and  conduct  system  level  tests  on  the  equipment  integrated  into  the 
system.  • 

Contractors  and  shipbuilders  are  instructed  to  install  GFE  in 
ships.      Often  the  equipment  to  be  installed  has  not  been  adequately 
modified  to  reflect  the  latest  change  status,    or  parts  are  missing. 
The  LBTS  has  not  escaped  this  either.      For  instance,    the  CWI  units 
for  the  Mk  92   FCS  were  received  without  several  cards  and/or  drawer  . 
components.      Often  such  equipment  is  installed  and  equipment /sub- 
system level  tests  are  in  progress  before  it  is  discovered  that  a  crystal 
buried  in  the  back  of  a  chassis  is  missing.      Cycling  high  risk  items 
through  the  LBTS  would  help  to  alleviate  this  problem. 

Another  concept  that  merits  consideration  encompasses  the 
cycling  of  equipment  through  the  LBTS  where  it  could  be  checked  out 
by  a  verification  and  test  team.     This  same  team  could  follow  the  sys- 
tem on  board  ship  where  they  would  be  responsible  for  integration 
testing.     Such  an  approach  would  definitely  reduce  test  time  and  since 
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the  follow  ships  are  scheduled  to  go  down  the  ways  at  the  rate  of  one 
per  month,    it  is  feasible  from  a  manning  standpoint. 

13.       How  were  the  following  site  selection  criteria  decided? 

-Cost  effectiveness  (stressing  maximum  utilization  of  existing 
assets  consistent  with  the  mission  of  the  facility  employed)? 

Once  Sperry  received  the  contract  to  design,   build  and  operate 
the  PFCSLBTS,    the  decision  as  to  where  the  test  site  should  be  located 
was  Sperry's.      Thus  cost  effectiveness  has  to  be  viewed  from  Sperry's 
viewpoint  and  this  information  is  not  available. 
-Minimum  MILCON  requirement. 
It  is  understood  that  one  of  the  factors  leading  to  the  decision 
to  place  Sperry  under  contractual  obligation  to  build  the  test  site  with 
R&D  funds  was  the  long  lead  time  required  to  obtain  funding  via  MIL- 
CON  channels.      MILCON  funding  requirements  were  thus  minimized 
(zero),   but  whether  or  not  a  greater  amount  of  R&D  funds  were  required 
to  erect  the  LBTS  is  unknown. 

-Adequate  source  of  professional  talent  and  skilled  labor,    good 
housing,    and  community  services. 

Once  again  such  considerations  were  made  by  Sperry.     It  is 
understood  that  considerations   such  as  those  listed  above  came  to  bear 
in  Sperry's  decision,    but  again,    this  information  is  not  available. 
-Availability  of  fleet  computer  programming  groups. 
It  does  not  appear  that  this  was  a  consideration.     UNIVAC  is 
tasked  with  the  development  of  programs,    and  Sperry  with  the  running 
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and  check  out  of  these  programs.     While  Navy  fleet  computer  groups 
are  monitoring  the  progress  and  will  become  actively  involved  in  the 
certification  of  the  PFCS,    it  does  not  appear  that  their  availability  was 
a  factor  in  site  selection. 

-Availability  of  fleet  units  in  the  area. 
This  was  not  a  consideration  as  can  best  be  determined. 

-Availability  of  controlled  Navy  aircraft  services  with  minimum 
airspace  restriction. 

Sperry  has  previously  used  the  area  in  which  the  PFCSLBTS  is 
located  to  conduct  tests  on  the  AN/SPG-55B  Terrier  FCS.     Basedon  - 
this  experience  it  was  postulated  that  adequate  aircraft  services  could 
be  obtained  to  support  PFCS  tests.     It  is  obvious  that  more  realistic 
flight  paths   could  be  programmed  if  the  LBTS  were  remotely  located, 
and  this  is  another  tradeoff  that  was  made  in  site  selection. 

-Easy  accessibility  coupled  with  good  security. 
Again  this  was  a  Sperry  decision.      Under  the  terms  of  their 
contract  Sperry  is   responsible  for  providing  security  at  the  LBTS. 
No  fault  can  be  found  with  their  system  which  consists  of  a  badge  and 
sign  in/out  log.      Visitors  must  be  signed  in  by  site  personnel  and 
escorted.      The  site  itself  is  easy  to  reach  and  more  than  adequate 
parking  is  provided. 

-Near  large  body  of  water  (for  electromagnetic  propagation 
considerations). 

There  is  no  problem  in  this  area  on  Long  Island. 
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-Controlled  electromagnetic  environment. 

Sperry  conducted  many  ECM/ECCM  checks  on  the  AN/SPG- 
55B  FCS.     While  the  environment  around  the  LBTS  can  be  controlled 
only  to  a  limited  extent,    it  must  be  remembered  that  the  radars  asso- 
ciated with  fire  control  systems  operate  in  a  frequency  range  not 
normally  employed  by  civilian  activities.     As  such  then,    little  inter- 
ference with  FCS  ECM/ECCM  testing  is  anticipated. 

-Accessibility  to  collocated  NAVSEC  engineering  groups  and 
SHAPM. 

Unless  the  LBTS  was  located  in  Washington,    the  present 
location  is  as  good  as  might  be  expected.     It  is  located  a  stone's  throw 
from  MacArthur  Airport  and  airline  schedules  to  and  from  Washington 
are  such  that  people  coming  to  the  LBTS  from  Washington  can  arrive 
in  "the  morning  and  depart  in  the  afternoon  with  a  minimum  of  delay. 
14.       How  will  the  test  site  be  certified?     What  does  certification 
of  the  test  site  mean?     Are  the  simulations  being  used  certified? 

The  LBTS  will  be  certified  by  a  certification  activity  desig- 
nated by  PMS-399.     At  the  present  time  (March  1974)  it  looks  like  this 
activity  will  be  the  eastern  detachment  of  the  Naval  Ships  Missile 
System  Engineering  Station. 

Certification,    as  defined  within  PF  Programs,    is  the  process 
of  conducting  a  test  procedure  at  the  PFCSLBTS  (or  other  facility)  in 
order  to  ensure  that  it  is  correct,    accurate  and  complete  with  respect 
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to  the  equipment  being  tested.     A  test  procedure  will  be  certified 
as  being  correct  on  satisfactory  completion  of  certification.      Certifica- 
tion of  the  PFCSLBTS  in  essence  means  that  the  site  has  been  tested 
as  a  combat  system  and  the  results  of  these  tests  show  that  the  site 
does  indeed  operate  as  the  combat  system  designed. 

Simulations  being  used  at  the  LBTS  at  the  present  time  do  not 
have  a  certification  in  the  sense  of  acceptance  via  testing  in  accordance 
with  an  approved  test  plan.     Sperry  is  developing  a  plan  to  demonstrate 
that  in  fact  the  simulations   represent  actual  equipment.     In  the  sense 
that  individual  simulations  support  the  overall  conduction  of  certifica- 
tion tests,    the  simulators  will  be  certified  at  the  time  the  PFCSLBTS 
is  certified. 

15.  What  factors  will  be  introduced  during  testing  to  simulate 
power  fluctuations,  power  failures,  heat  and  cold,  air  conditioning 
failures,    chill  water  that  isn't  chilled,    etc.  ? 

None.      Environmental  testing  to  required  specifications  is  to 
have  been  completed  during  equipment  manufacture  and  test.     Deliberate 
failures  are  not  planned  to  be  introduced  during  the  course  of  testing  at 
the  LBTS.      The  site  is  powered  from  the  Long  Island  Power  Company, 
and  this  power  is  thought  to  be  less  regulated  and  more  apt  to  drop 
than  shipboard  power.     There  will  probably  be  a  sizeable  number  of 
other  type  failures  encountered  in  the  normal  course  of  events. 


120  ' 


In  summary,    this  section  attempts  to  pose  some  pointed  questions 
concerning  the  PFCSLBTS,   and  to  provide  some  answers.      Life  being 
what  it  is,    there  always  appears  to  be  more  questions  than  answers. 
Much  was  left  unquestioned  and  unanswered  such  as:     "What  is  the 
value  and  what  has  been  said  and  done  about  the  necessity  of  developing 
and  retaining,    permanently,    within  the  Navy,    the  expertise  required  to 
integrate  a  combat  system?  ";   "How  will  the  LBTS  be  integrated  with 
BIS  and  OPTEVFOR  for  lead  and  follow  ship  testing?  "  and  "What  are 
the  real  cost/effectiveness  considerations  for  the  LBTS?  " 
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VIII.     CONCLUSION  AND  AREAS  FOR  FURTHER  CONSIDERATION 

A.  GENERAL 

This  thesis  should  be  considered  neither  a  compendium  of  combat 
system  test  and  evaluation  nor  a  handbook  of  solutions  based  on  the 
approach  of  the  Patrol  Frigate  program  via  a  land-based  test  site. 
It  does  trace  emerging  doctrines  encompassing  ship  acquisition  T&cE 
and  investigates  the  PFLBTS  as  an  innovative  approach  to  a  practical 
solution.      The  complexities  of  modern  shipboard  combat  systems  have 
been  discussed  as  has  efforts  to  formalize  the  test  and  evaluation 
requirements  for  these  systems.      To  date  however,    the  strides  that 
have  been  made  in  T&E  are  due  largely  to  the  confiction  of  the  managers 
involved.      The  use  of  land-based  sites  in  T&E  investigated  in  this  thesis 
has  been  developed  from  an  empirical  rather  than  a  theoretical  analysis. 
Successful  implementation  of  T&E  policy  must  be  based  primarily  on 
past  experience,    and  personnel  involved  with  ship  T&E  policy  should 
have  some  background  in  ship  acquisition  programs  both  from  a  the- 
oretical and  a  "hands  on"  base.      This  will  allow  thorough  attention  to 
proposals  for  future  improvements  that  will  continually  be  present  in 
this  emergent  field. 

B.  SIMULATION  AT  THE  PFLBTS 

In  recognition  of  the  cost  to  procure,    install,    operate,    and  main- 
tain a  total  ship  combat  system,    including  all  its  sensor  and  weapons 
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subsystems,    the  PFLBTS  relies  on  simulation  for  interface  testing  of 
certain  hardware  and  software  items.      This  imposes  limitations  on  the 
facility;  these  limitations  appear  acceptable  in  that  the  PFLBTS  has 
consistently  retained  the  option,    within  funding  constraints,    for  instal- 
lation of  further  equipment  and  subsystems  should  experience  prove  it 
to  be  vital  to  the  T&E  program. 

Simulations  provide  T&E  personnel  the  opportunity  to  subject  men 
and  machines  to  a  much  wider  variety  of  test  situations  than  one  could 
normally  expect  when  working  with  complete  systems. 

C.      LBTS  COST/EFFECTIVENESS  CONSIDERATIONS 

When  considering  the  expenditure  of  funds  required  for  the  estab- 
lishment and  operation  of  a  LBTS,    it  should  be  remembered  that,    with 
few  exceptions,    these  costs  will  be  incurred  anyway,    i.  e.  ,    if  not  at  the 
LBTS,    then  somewhere  else.     The  task  of  proving  the  integration  of  the 
total  ship  combat  system  must  necessarily  be  accomplished. 

Although  the  "effectiveness"  of  a  test  facility  is  not  amenable  to 
precise  definition  or  measurement,    its  cost  can  be  defined  and  es- 
timated with  some  certainty.     If  alternatives  in  execution  of  a  TSTP 
are  limited  to  T&E  at  a  LBTS  or  T&E  aboard  ship,    then  both  alter- 
natives could  by  definition  be  "effective"  and  thus  have  a  constant 
and  equal  effectiveness,    i.  e.  ,    they  are  of  "equal  benefit"  in  economic 
analysis  terms. 
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As  previously  stated,  the  costs  of  the  LBTS  itself  can  be  deter- 
mined with  some  degree  of  precision.  The  cost  of  obtaining  a  success- 
ful T&E  via  fragmented  testing  at  a  diversity  of  test  sites  under 
different  managers  may  only  be  conjured  up.  Based  on  the  experiences 
of  past  T&E  programs  such  as  the  DLGN-36,  it  is  postulated  that  ship- 
board testing,  having  as  its  basis  adequate  LBTS  support,  is  much  less 
costly.      The  answer  is  locked  in  the  future. 

D.      ADVANTAGES  OF  LAND-BASED  TESTING 

Some  advantages  of  utilization  of  a  dedicated  LBTS  such  as  that 
being  employed  in  the  PF  Program  include: 

1.  Software  debug  and  testing  can  be  accomplished  in  a  controlled 
hardware  environment. 

2.  System  integration  debugging  can  be  started  earlier  since  it 
is  not  tied  to  the  ship  construction  schedule  and,    parenthetically,    con- 
struction delays. 

3.  Software  debugging  can  be  accomplished  in  an  environment 
free  of  industrial  work. 

4.  Use  of  simulation  and  simulators  can  be  optimized  both  on  a 
single  project  and  multi-project  levels.      One  simulator  can  support 
more  than  one  system  integration  effort  as  can  hardware. 

5.  A  single  integration  facility  eliminates  redundant  test 
equipment. 
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6.  From  the  "people"  perspective  it  concentrates  your  best 
people  at  one  site.      These  are  the  same  people  that  follow  the  system 
integration  effort  through  to  final  ship  acceptance. 

7.  Support  software,    debug  aids,    data  extraction  and  reduction 
and  analytical  tools  can  be  provided  on-site  and  shared  by  more  than 
one  project. 

8.  Test  site  provides  earlier  "hands  on"  experience  for  the 
nucleus  crew  and  key  personnel. 

9.  GFE  software  can  be  "certified"  prior  to  delivery  to  the 
contractor  shipbuilder. 

10.  The  test  site  is  a  step  toward  satisfying  the  necessity  to 
develop  and  retain,    permanently,    within  the  Navy,    the  expertise 
required  to  test,    evaluate,    and  integrate  a  ship  combat  system. 

11.  It  meets  DOD  requirements  to  "fly-before-buy.  " 

E.   T&E  AND  THE  PATROL  FRIGATE 

Admiral  Woodfin  and  others  have  cited  several  reasons  why  the 
PFCS  integration  should  not  proceed  down  the  "primrose  path"  taken 
by  previous  ship  acquisitions.      The  PF  Program  has  integrated  total 
ship  testing  into  the  design  from  the  outset.      This  includes  HME, 
combat  system,    and  mobility.     The  integrated  test  program  includes: 
receipt  inspection,    installation  and  check  out  of  equipment,    equipment 
testing,    intra  system  tests,    inter  system  tests,    builder  trials,    and 
acceptance  trials.      Integration  includes  utilization  of  a  dedicated 
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CSLBTS  that  is  a  faithful  reproduction  of  the  ship.     The  computer  soft- 
ware,   combat  system  integration,    and  test  site  development  is  being 
performed  by  one  contractor.     The  tests  performed  at  the  test  site 
will  be  directly  transferable  to  the  ship  and  test  site  personnel  will  be 
involved  in  the  shipboard  testing.      The  PF  Program  has  a  combat  sys- 
tems integration  manager,    responsible  for  the  total  system.      Finally, 
the  lead  ship  is  being  acquired  on  a  cost  type  contract  basis,    which 
will  facilitate  dealing  with  integration  problems  as  they  occur. 

In  conclusion,    although  there  is  little  in  major  weapon  system 
acquisition  that  can  be  predicted  with  absolute  certainty,    upon  careful 
consideration,    few  can  deny  the  legitimacy  and  fertility  of  the  utiliza- 
tion of  a  LBTS  to  reduce  uncertainty  in  ship  acquisition  testing.      "By 
your  test  results,    ye  shall  be  known",    could  well  serve  as  a  motto  for 
the  PFCSLBTS. 

F.      AREAS  FOR  FURTHER  CONSIDERATION 

1.       During  the  time  frame  that  this  thesis  was  being  researched 
and  written,    the  PFCSLBTS  was  in  the  process  of  being  built.     All  of 
the  equipment  was  not  scheduled  to  have  arrived,    and  the  bulk  of  the 
equipment  at  the  site  was  in  the  process  of  being  powered.     Subsystem 
and  system  testing  obviously  had  not  started.     It  would  be  of  interest 
to  continue  to  follow  the  evolution  of  the  PFCSLBTS  through  to  its 
final  wake,    be  it  at  the  end  of  DSARC  III,    or  when  the  last  PF  is 
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decommissioned  some  thirty  years  or  so  hence.     Main  points  for 
consideration  include: 

-Did  the  PFCSLBTS  fill  its  mission  to  provide  the  facilities  and 
personnel  to  test  integrated  ship  combat  systems. 

-How  effective  was  the  LBTS  in  discovering  and  correcting  (or 
causing  correction),    subsystem,    interface  and  computer  program 
problems. 

-What  was  the  post  audit  cost  of  the  test  site  and  did  indeed  a 
constant  effectiveness  situation  exist  between  the  LBTS  and  other 
viable  alternatives. 

2.  The  use  of  a  LBTS  and  of  prototyping  has  been  proposed  as  an 
alternative  conducive  to  reducing  uncertainty  in  an  acquisition  program. 
A  subjective  analysis  of  available  alternatives  leads  one  to  the  con- 
clusion that  a  LBTS  is  a  viable  approach.     Would  the  same  results 

be  obtained  from  a  quantitative  analysis? 

3.  Gordon  W.    Rule  in  a  recent  newspaper  article,    advocated 
slowing  the  production  schedule  of  the  PF  to  gain  additional  time  for 
testing  of  the  Mk  92  FCS,    a  Dutch  designed  system,    and  the  Mk  75  gun, 
an  Italian  design.     Another  article  reported  that  the  cost  of  the  PF  is 
now  about  $70  million  each,    compared  with  $45  million  in  1971.     It 
would  be  of  interest  to  analyze  actual  costs  with  budgeted  costs  of  the 
PF  Program,    tempered  by  inputs   such  as  Mr.    Rule's. 


127 


APPENDIX  A 
TEST  MODEL  FOR  ACQUISITION  CYCLE 


t/» 


S  £«» 

if! 


*  «  ■ 

-  •-  X 


s " » s :  r 

5  •  s  •■  r 


*-     3  «  •-        b  •  »  x  »-  a  4 


si* 

B  =  S 

ISi 

»  z,  > 

Hi 

■i 


JS 


.~s 


S    f 


s: 


S  S  8 


«o< 


*o< 


o 


>     W    z 

H    M    O 

a  a  u 


*0< 


►-    m    M 

S>    m 

U    Q     3 


m 

z  e 
%  « 

_i   at 
■I   ►-    K 

>  z  3 

ta    O     v. 
QUO 


«o 


>  » 

O     3 


* 

5 

0£ 

■* 

I 

>- 

z 
o 

o 

U 

**> 

Ml 

kf 

X 

»- 

o 

z 

6 

< 

kJ 

S 

7 

5 

»- 

u 

X 

«u 

w 

f  > 

< 

•m 

©1 

< 

u 

H 

<oo 


Si 
ii 


x    3 


o    < 

5  S 


3 


o 

X 


M 
X 


o    "* 
X     ~ 

>-  " 


128 


APPENDIX  B 
THE  PF  COMBAT  SYSTEM/SPERRY  TEST  CENTER 
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